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This  study  examines  how  teachers’  qualifications  and  pedagogical  approaches  differ 
across  the  35  National  Guard  Youth  Challenge  Program  (ChalleNGe)  sites.  ChalleNGe 
serves  high  school  dropouts  with  both  academic  and  noncognitive  components, 
including  a  postresidential  mentoring  phase.  We  developed  and  fielded  an  online 
teacher  survey.  Using  the  survey  data  and  information  from  the  programs’  annual 
reports,  we  investigate  whether  different  teacher  qualifications  or  pedagogical 
approaches  are  correlated  with  programs’  average  cadet  outcomes  (e.g.,  graduation 
rates,  postresidential  placement).  We  also  attempt  to  identify  classroom  methods 
that  may  be  effective  with  disadvantaged  youth,  who  typically  perform  below  grade- 
level  (the  ChalleNGe  student  body).  Of  the  numerous  factors  we  considered,  only  a 
few  were  found  to  be  significantly  correlated  with  cadet  outcomes— namely,  the 
prevalence  of  postgraduate  degrees  among  teaching  staff,  consistency  in  the  gender 
makeup  of  classes,  a  greater  emphasis  on  small-group  instruction,  emphasis  on 
advanced  math  and  “math  life  skills,”  and  a  focus  on  critiquing  and  evaluating  texts 
and  writing  to  summarize. 
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Executive  Summary 


Background 

The  National  Guard  Youth  Challenge  Program  (ChalleNGe)  is  a  quasi-military,  22- 
week  residential  program  designed  to  serve  16-  to  18-year-old  high  school  dropouts 
as  well  as  students  at  risk  of  dropping  out  (students  who  have  earned  far  fewer 
credits  than  expected  are  considered  at  risk  of  dropping  out).  The  program  also 
includes  a  12-month  postresidential  mentoring  component.  During  this  time,  cadets 
and  their  mentors  are  asked  to  report  back  to  the  program  regarding  whether  the 
cadet  is  employed,  in  school,  or  serving  in  the  military.  Currently,  ChalleNGe  is 
offered  at  35  locations  in  29  states  and  Puerto  Rico. 

Many  cadets  arriving  at  ChalleNGe  have  lacked  guidance  or  leadership  throughout 
their  adolescent  years,  which  may  have  contributed  to  their  struggles  in  the 
traditional  high-school  environment.  The  ChalleNGe  staff  will,  for  some  cadets,  be 
the  first  positive  role  models  they  have  encountered  in  quite  some  time.  In  this  light, 
the  ChalleNGe  teachers,  with  whom  the  cadets  interact  frequently  and  for  extended 
periods  of  time,  are  important  to  cadets’  overall  success— not  only  academically  but 
also  in  the  growth,  maturation,  and  development  of  trusting  relationships  that  occur 
at  ChalleNGe. 

Historically,  however,  little  has  been  known  about  the  ChalleNGe  teachers’ 
characteristics,  teaching  philosophies,  or  pedagogical  methods— neither  regarding 
the  program  as  a  whole  nor  in  terms  of  program-level  variation.  As  a  result,  it  was 
not  possible  to  evaluate  whether  any  differences  in  the  programs’  teachers  (in  terms 
of  either  qualifications  or  teaching  strategies)  had  lasting  implications  for  cadets. 
With  this  study,  we  have  begun  to  fill  that  void. 

The  main  contribution  of  this  paper  is  to  document  differences  in  the  composition  of 
teaching  staffs  across  ChalleNGe  programs.  Secondarily,  we  attempt  to  determine  if 
any  variation  in  teachers’  qualifications  or  pedagogical  approaches  across  ChalleNGe 
programs  are  statistically  significantly  associated  with  cadets’  average  outcomes. 


iii 


Approach 


CNA 

a  ^rw  t  'TiriN^? 


Because  ChalleNGe  had  no  preexisting  information  about  its  teachers,  tackling  these 
issues  first  required  the  collection  of  primary  data.  To  do  this,  we  administered  an 
online  survey  to  all  ChalleNGe  teachers.  The  survey  asked  about  their  qualifications 
(including  teaching  certification  or  license,  college  majors,  highest  educational 
degree  attained,  and  previous  teaching  experience),  their  typical  class  size  and 
students’  characteristics,  and  their  pedagogical  approaches  and  teaching 
philosophies.  We  further  asked  the  math  and  English/language  arts  teachers  a  series 
of  more  detailed  questions,  since  these  subjects  are  generally  considered  the  two 
core  curricular  areas. 

Since  our  outcome  variables  are  average  measures— e.g.,  a  program’s  average  2014 
graduation  rate  or  average  2014  improvement  in  the  Test  of  Adult  Basic  Education 
(TABE)  score— we  are  limited  to  running  program-level  regressions  (with  a  sample 
size  of  34)  and  we  therefore  collapse  all  teacher-level  information  into  program 
averages.  A  few  important  caveats  regarding  our  analysis  follow: 

1.  We  cannot  match  teachers  to  particular  cadets  because  all  cadets  are  under  the 
purview  of  all  teachers  at  most  ChalleNGe  programs. 

2.  We  do  not  have  postresidential  outcome  data  on  all  cadets.  The  ChalleNGe 
programs  attempt  to  follow  up  with  cadets  1,  6,  and  12  months  after 
graduation,  but  not  all  cadets  are  reachable  and  some  do  not  respond.  This 
sample  of  cadets  may  suffer  from  selection  bias,  which  could  result  in  either 
understating  or  overstating  any  effects  on  postresidential  outcomes. 

3.  We  calculated  postresidential  placement  rates  by  dividing  the  number  placed 
by  1,  6,  or  12  months  by  the  total  number  of  graduates  in  that  class.  It  would 
be  more  accurate  to  divide  by  the  number  of  cadets  who  provided 
postresidential  placement  information,  but  this  information  is  not  available. 

4.  The  most  recent  postresidential  data  available  are  for  previous  classes. 
Therefore,  our  findings  regarding  postresidential  outcomes  will  be  least 
accurate  for  those  programs  that  have  experienced  significant  teacher  turnover 
since  the  postresidential  cadets  in  the  2014  reports  graduated. 

With  these  caveats  in  mind,  we  summarize  our  most  notable  findings. 
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The  wide  range  of  questions  we  asked  the  ChalleNGe  teachers  and  the  resulting 
volume  of  information  we  collected  produced  a  wide  range  of  findings.  A  few  are 
particularly  noteworthy.  First,  the  ChalleNGe  teacher  population  as  a  whole  is  a  well- 
qualified  teaching  force.  There  is,  of  course,  variation  from  one  program  to  the  next, 
but  overall,  45  percent  of  ChalleNGe  teachers  hold  more  than  a  bachelor’s  degree, 
and  the  average  teacher  has  14  years’  teaching  experience  and  5  years’  experience 
teaching  at  ChalleNGe.  Thus,  the  average  cadet  is  being  taught  by  staff  who  are  not 
only  well  educated  but  also  have  significant  classroom  experience.  In  addition,  with 
an  average  of  5  years  of  ChalleNGe  teaching  experience,  these  teachers  will  not  be 
fazed  by  the  behavioral  and  attention  problems  likely  to  emerge  in  a  ChalleNGe 
classroom;  they  are  prepared  to  deal  with  such  issues  effectively. 

We  also  found  significant  variation  in  teachers’  pedagogies  across  ChalleNGe 
programs— including  how  cadets  are  assigned  to  classes,  whether  male  and  female 
cadets  are  separated  or  taught  in  mixed-gender  classrooms,  how  instructors  divide 
their  time,  and  the  amount  of  homework  assigned  in  a  typical  week.  Only  some 
elements  of  teachers’  pedagogies,  however,  correlate  with  average  cadet  outcomes. 
Four  particular  findings  stand  out  as  having  potential  policy  implications: 

•  Consistency  in  the  male/female  composition  of  the  classroom  matters. 
Regardless  of  whether  the  classes  are  always  gender  mixed  or  always  gender 
separated,  either  alternative  led  to  better  cadet  outcomes  than  having  some 
classes  with  mixed  gender  and  some  separated. 

•  Cadets’  1 -month  placement  rates  are  positively  and  significantly  correlated 
with  the  amount  of  small-group  and  individual  instruction  (versus  whole 
group). 

•  Cadets  benefit  from  greater  emphasis  on  advanced,  rather  than  basic,  math 
topics  and  “math  life  skills.” 

•  The  specific  types  of  writing  and  reading  assigned  in  English/language  arts 
classes  make  a  difference. 
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Introduction 


The  National  Guard  Youth  Challenge  Program  (ChalleNGe)  is  a  quasi-military,  22- 
week  residential  program  designed  to  serve  16-  to  18-year-old  high  school  dropouts 
as  well  as  students  at  risk  of  dropping  out  of  school  (students  who  have  earned  far 
fewer  credits  than  expected  are  considered  at  risk  of  dropping  out).  The  program 
also  includes  a  12-month  postresidential  mentoring  component.  During  this  time, 
cadets  and  their  mentors  are  asked  to  report  back  to  the  program  regarding  whether 
the  cadet  is  employed,  in  school,  or  serving  in  the  military.  Currently,  ChalleNGe  is 
offered  at  35  locations  in  29  states  and  the  territory  of  Puerto  Rico. 

The  ChalleNGe  model  includes  both  academic  and  noncognitive  components,  such  as 
leadership/followership,  responsible  citizenship,  service  to  community,  life-coping 
skills,  physical  fitness,  health  and  hygiene,  job  skills,  and  academic  excellence.  The 
academic  program  is  administered  in  a  manner  similar  to  that  found  in  a  traditional 
high  school  setting.  Teachers  are  given  curricular  guidelines  regarding  topics  they 
must  cover  but  otherwise  have  some  latitude  regarding  classroom  management  and 
pedagogical  methods. 

Programs  publicize  and  recruit  by  advertising,  by  building  relationships  with  a 
variety  of  people  who  come  in  contact  with  at-risk  youth,  and  by  word  of  mouth. 
Each  program  typically  receives  more  applications  than  it  can  accommodate  during 
each  class  (each  program  has  two  cadet  classes  per  fiscal  year).  Applicants  may  be 
denied  admission  because  of  ineligibility  or  due  to  space  or  funding  limitations.^ 
Programs  do  not,  however,  deny  admission  based  on  test  scores  or  any  other 
academic  requirement. 

The  primary  academic  goal  of  most  ChalleNGe  programs  is  for  cadets  to  pass  the 
General  Educational  Development  (GED)  tests.  Some  programs,  however,  award 
alternate  credentials,  such  as  state  high  school  diplomas,  to  cadets  who  complete  the 
program.  Still  other  programs  focus  on  credit  recovery,  allowing  cadets  to  reenroll  in 
and  graduate  from  their  previous  high  school  after  completing  ChalleNGe.  Einally, 
some  programs  are  considered  schools  and  award  regular  high  school  diplomas. 
These  differences  are  a  function  of  state-level  requirements  and  how  the  programs 
choose  to  operate. 


^  Eligibility  requirements  can  be  found  at  https://www.iointservicessupport.org/NGYCP.  They 
include  being  a  citizen  or  legal  resident  of  the  United  States,  being  free  from  use  of  illegal 
drugs  or  substances,  and  being  physically  and  mentally  able  to  participate  in  the  program. 
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Teachers  at  ChalleNGe  programs  come  from  a  wide  variety  of  backgrounds;  some  are 
certified  teachers,  while  others  are  not.  Some  ChalleNGe  teachers  have  substantial 
experience  in  traditional  classrooms,  some  come  from  a  background  of  working  with 
disadvantaged  youth,  and  some  have  taught  primarily  in  GED  preparatory  programs 
or  other  adult  education  programs.  The  classroom  setup  varies  across  ChalleNGe 
programs  as  well,  since  it  is  left  to  the  programs’  discretion.  At  some  programs, 
teachers  specialize  in  a  single  subject;  at  others,  teachers  work  with  the  same  group 
all  day,  covering  all  subjects.  Finally,  most  classrooms  are  single-sex,  but  some  are 
not. 

While  passing  the  GED  or  obtaining  a  high  school  (HS)  diploma  or  alternate  credential 
is  considered  the  ultimate  goal,  programs  also  strive  to  help  cadets  make  as  much 
academic  progress  as  possible  during  the  residential  phase.  To  gauge  cadets’ 
progress,  the  ChalleNGe  programs  administer  a  standardized  test  (the  Test  of  Adult 
Basic  Education,  or  TABE)  at  the  beginning  and  end  of  the  program.  Academic  gains 
vary  by  location  and  by  class.  For  the  FY14  ChalleNGe  classes,  TABE  gains  range  from 
0.1  to  4.3  (e.g.,  a  score  of  4.3  indicates  performance  at  the  third  month  of  fourth 
grade).  Some  of  this  variation  is  likely  due  to  differences  in  the  academic  abilities  of 
cadets  when  they  enter  the  program.  Gains  are  more  difficult  with  cadets  at  higher 
academic  levels,  for  example,  because  there  is  less  room  for  “growth.”  It  is  also 
difficult  to  realize  large  academic  gains  with  cadets  who  arrive  at  ChalleNGe 
performing  significantly  below  grade  level.  Although  cadets’  incoming  academic 
abilities  (and  the  resulting  achievable  TABE  gains)  are  factors  beyond  the  ChalleNGe 
program’s  control  (since  there  are  no  academic  restrictions  on  admissions),  some  of 
the  variation  observed  in  cadets’  TABE  gains  may  be  due  to  differences  in  the 
pedagogical  methods  used  by  ChalleNGe  teachers.  In  this  case,  modifications  to 
classroom  teaching  could  yield  larger  academic  gains. 

In  traditional  classrooms,  the  influence  of  pedagogy  on  student  outcomes  is  fairly 
clear,  while  the  influence  of  a  teacher's  characteristics,  education,  and  experience  is 
far  less  clear. ^  Although  a  broad  body  of  literature  addresses  student  outcomes, 
virtually  no  research  exists  tying  the  elements  of  classroom  preparation  to  GED 
success.  This  is  mainly  because  most  GED  test-takers  undertake  only  minimal 
preparation,  and  usually  not  in  a  classroom  setting  [4].  Although  there  is  evidence 
that  higher  GED  test  scores  are  tied  to  better  labor  market  outcomes,  very  little  is 
known  about  what  kind  of  preparation  is  required  for  young  GED  test-takers  to 
obtain  these  higher  GED  scores. 


^  For  discussions  of  classroom  instruction  and  student  outcomes,  see  (along  with  many  others) 
Swanson  and  Stevenson  (2002)  [1]  and  Tarr  et  al.  (2008)  [2].  The  first  reference  is  particularly 
relevant  because  our  surveys  include  some  of  the  same  measures  of  classroom  pedagogy  and 
practice  as  the  National  Assessment  of  Educational  Progress  (NAEP)  surveys.  For  some  of  the 
disparate  results  on  teacher  characteristics  and  student  outcomes,  see  The  Handbook  of  the 
Economics  of  Education  [3]. 
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This  study  investigates  how  pedagogical  approaches  differ  across  programs  and 
whether  these  differences  correlate  with  cadet  outcomes.  It  may  also  suggest 
classroom  methods  that  are  especially  effective  with  disadvantaged  youth  who 
perform  below  grade  level.  For  example,  past  research  shows  that  high  school 
students  performing  below  grade  level  are  most  likely  to  be  taught  by  "drill-and-kill" 
methods  (i.e.,  methods  focused  almost  exclusively  on  rote  memorization  rather  than 
learning  and  synthesizing  broader  concepts),  but  no  analysis  has  been  conducted 
regarding  whether  this  approach  is  effective  with  this  population.^  Finally,  we 
attempt  to  link  classroom  practices  to  the  longer  term  outcome  of  placement  after 
completing  the  program.  Thus,  the  main  contribution  of  this  paper  is  to  document 
differences  across  ChalleNGe  programs  in  the  composition  of  their  teaching  staff. 
Secondarily,  we  attempt  to  determine  if  any  variation  in  teachers'  qualifications  or 
pedagogical  approaches  across  programs  are  statistically  significantly  associated 
with  cadets’  average  outcomes. 


^  See,  for  example,  [5]. 
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Our  primary  data  source  is  an  online  survey,  administered  to  all  ChalleNGe  teachers 
via  email.  Teachers  were  asked  specifically  about  their  qualifications  and  previous 
teaching  experience  (within  and  outside  ChalleNGe),  their  typical  class  size  and 
students’  characteristics,  as  well  as  their  pedagogical  approaches  and  teaching 
philosophies.  In  addition,  math  and  language  arts  teachers  were  asked  a  series  of 
more  detailed,  subject-specific  questions  about  how  they  use  classroom  time  and  on 
what  concepts  they  focus.  The  survey  was  live  from  October  7,  2014,  through 
November  25,  2014.^  The  survey  was  designed  for  ChalleNGe  teachers  and  certain 
questions  are  specific  to  the  program.  To  understand  more  about  classroom 
practices  and  techniques,  however,  we  drew  from  a  number  of  established,  national 
surveys  such  as  [6],  [7],  [8],  [9],  and  [10]. 

Our  other  data  source  is  the  ChalleNGe  programs’  annual  reports  (supplied  by  each 
program  to  the  director  of  the  National  Guard  Youth  Challenge  Program  (NGYCP)). 
These  reports  contain  information,  by  class,  on  the  number  of  registered  cadets,  the 
number  that  ultimately  graduated,  the  cadets’  average  scores  on  the  Test  of  Adult 
Basic  Education,  and  their  postresidential  outcomes.  The  postresidential  data 
indicate  how  many  cadets  from  previous  classes  were  enrolled  in  school,  had  found 
employment,  or  had  enlisted  in  the  military  at  the  1-,  6-,  and  12-month  marks  (after 
their  ChalleNGe  graduation). 

Cadets  take  the  TABE  at  the  beginning  and  end  of  ChalleNGe.  The  TABE,  designed  for 
placement  of  adult  learners,  is  often  used  as  an  assessment  tool  in  adult  education 
programs  with  a  focus  on  passing  the  GED  tests.  Each  subsection  of  the  TABE  is 
scored  to  indicate  grade  level  (for  example,  a  score  of  9.3  indicates  performance  at 
the  third  month  of  ninth  grade).  We  use  the  overall  TABE  score  (formed  from 
averaging  scores  on  the  subtests).  The  subtests  are  Math,  Applied  Math,  Language, 
and  Reading.  The  Math  section  is  made  up  of  computational  problems  requiring  test- 
takers  to  perform  addition,  subtraction,  multiplication,  and  division;  to  work  with 
percentiles,  fractions,  and  exponents;  and  to  solve  basic  algebra  problems.  The 
Applied  Math  section  is  made  up  of  word  problems  requiring  the  following  abilities: 


^  The  initial  planned  survey  duration  was  shorter,  but  we  extended  it  in  an  effort  to  increase 
participation  since  teacher  participation  was  unexpectedly  low  early  on.  We  also  asked  the 
ChalleNGe  program  directors  to  encourage  their  teachers’  participation.  In  a  handful  of  cases, 
teachers  experienced  technical  difficulties  and  were  unable  to  access  the  survey  from  their 
systems;  we  mailed  paper  versions  of  the  survey  to  these  teachers. 
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chart  and  table  comprehension,  basic  equation  setup,  coordinate  graphing,  an 
understanding  of  some  limited  geometry,  and  application  of  the  concepts  of 
fractions,  percentiles,  and  algebra  in  the  context  of  word  problems.  The  Language 
section  Includes  questions  on  grammar  and  punctuation,  combining  sentences  to 
preserve  their  meanings,  and  some  basics  of  paragraph  composition.  The  Reading 
section  Involves  reading  passages  or  detailed  charts/tables  and  answering  questions 
about  the  content. 

The  intent  of  our  survey  and  analysis  is  twofold.  First,  we  aimed  to  collect 
information  not  previously  available  on  the  ChalleNGe  teacher  population  to  see  how 
teacher  qualifications,  pedagogical  approaches,  and  other  characteristics  vary  across 
programs.  This  analysis  rehes  largely  on  cross-tabulations  of  survey  results  across 
programs.  Second,  we  wanted  to  determine  whether  any  particular  characteristics— 
and,  thus,  any  particular  differences  across  programs— would  affect  cadets’  success. 
To  do  so,  we  use  a  number  of  measures  of  success:  program  graduation  rates; 
whether  cadets  have  been  “placed”  1,  6,  or  12  months  after  ChalleNGe  completion 
(placement  includes  school  emollment,  employment,  and  military  enlistment);  and 
cadets’  improvements  on  the  TABE.  We  use  regression  analysis  to  determine  which 
teacher  qualifications  and  pedagogical  methods  are  most  important  in  determining 
cadets’  average  outcomes.  Since  our  outcome  variables  are  average  measures  (e.g.,  a 
program’s  average  2014  graduation  rate  or  average  2014  TABE-score  improvement), 
we  are  limited  to  running  program-level  ordinary  least  squares  (OLS)  regressions 
(with  a  sample  size  of  34),  and  we  collapse  all  teacher-level  information  into  program 
averages  (e.g.,  average  years  of  teaching  experience).^ 

We  note  a  few  important  caveats  regarding  our  analysis  of  how  teacher  quahflcations 
and  pedagogical  approaches  are  correlated  with  cadets’  outcome  variables.  First,  we 
cannot  match  teachers  to  particular  cadets  because  all  cadets  are  under  the  purview 
of  all  teachers  at  most  ChalleNGe  programs.  Each  program,  in  general,  has  only  one 
science  teacher,  one  math  teacher,  one  English  teacher,  and  one  social  studies 
teacher;  therefore,  all  cadets  take  classes  from  all  teachers.  The  fact  that  we  have  no 
individual-level  cadet  performance  data  (which  we  could  theoretically  match  to 
specific  teachers)  does  not  matter  since  all  cadets  would  be  matched  to  all  teachers. 

As  a  result,  we  use  a  program’s  average  graduation  rate,  TABE  improvement,  and 
postresidential  placement  rates  as  our  outcome  variables  and  use  each  program’s 
average  teacher  characteristics  and  pedagogical  methods  as  our  regressors.  Thus,  we 
are  able  to  answer  only  questions  about  how  programs’  average  outcome  variables 
change  hi  response  to  corresponding  changes  in  then  average  teacher  population.  By 


^  Although  not  presented  in  our  tables  and  findings  throughout  this  report,  the  TABE 
improvement  regressions  also  contain  a  control  for  cadets’  average  pre-TABE  scores.  As 
expected,  this  variable  has  a  negative  and  significant  impact  on  TABE  improvement  over  the 
course  of  ChalleNGe.  The  higher  the  overall  TABE  scores  with  which  cadets  enter  ChalleNGe, 
the  smaller  their  score  improvement  can  be,  since  there  is  a  maximum  score  they  can  attain. 
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using  teachers’  average  characteristics  and  pedagogical  approaches  at  a  particular 
program,  we  are  indirectly  assuming  that  the  distribution  of  teachers  across  classes 
is  identical  (i.e.,  that  all  cadet  classes  within  a  program  are  taught  by  teachers  with 
the  same  characteristics  and  who  use  the  same  approaches).  These  estimates  would 
be  more  precise  if  we  were  able  to  identify  the  specific  students  taught  by  each 
teacher;  we  could  then  correlate  individual  cadets’  outcomes  with  individual 
teachers’  characteristics.'* 

Second,  we  do  not  have  postresidential  outcome  data  on  all  cadets.  The  ChalleNGe 
programs  attempt  to  follow  up  with  cadets  1,  6,  and  12  months  after  graduation,  but 
not  all  cadets  are  reachable  and  some  likely  simply  do  not  respond.  Our 
postresidential  analysis,  clearly,  is  limited  to  those  cadets  the  programs  were  able  to 
contact  who  were  willing  to  provide  the  requested  information.  It  is  possible  that  this 
sample  of  cadets  suffers  from  selection  bias— that  is,  that  those  who  provide  their 
postresidential  statuses  are  systematically  different  from  those  who  do  not,  which 
could  result  in  either  an  understatement  or  overstatement  of  any  effects  on 
postresidential  outcomes.  In  addition,  the  placement  rates  we  use  are  calculated  by 
dividing  the  number  placed  by  1,  6,  or  12  months  by  the  total  number  of  graduates 
in  that  class.  It  would  be  more  accurate  to  divide  the  number  of  cadets  placed  by  the 
number  of  cadets  who  provided  postresidential  placement  information.  The  number 
of  cadets  providing  this  information,  however,  is  not  available.  Our  placement  rates 
(number  placed/number  graduated),  therefore,  are  likely  to  underestimate  the  true 
placement  rates,  since  cadets  who  placed  but  did  not  respond  to  the  postresidential 
surveys  are  not  included.  In  fact,  the  only  scenario  in  which  our  placement  rates  are 
not  underestimates  is  if  all  program  graduates  who  were  placed  responded  to  the 
survey  and  only  those  were  not  placed  did  not  respond.  This  seems  unlikely. 

Finally,  the  most  recent  postresidential  data  available  are  for  previous  classes.  This 
information  was  extracted  from  the  2014  annual  reports;  the  postresidential 
information  provided  in  the  2014  reports  pertains  to  cadets  who  graduated  at  least 
12  months  ago.  Thus,  the  teachers  who  responded  to  our  survey  (administered  in 
October  and  November  of  2014)  may  not  be  the  teachers  responsible  for  the 
instruction  of  those  cadets  whose  postresidential  outcomes  are  captured  in  the  2014 
annual  reports.  This  is  not  only  because  we  cannot  be  certain  which  teachers  taught 
which  cadets,  but  also  because  teachers  who  taught  the  earlier  cadets  may  no  longer 
be  at  the  programs  (and  thus  not  be  in  our  survey  population).^  Our  findings 


•*  In  addition,  because  we  lack  data  on  those  who  did  not  respond  to  our  survey,  we  are  forced 
to  assume  that  the  average  responder  has  similar  characteristics  and  pedagogies  as  the  average 
nomesponder.  Without  data  on  nomesponders,  it  is  impossible  to  know  how  significantly  our 
results  are  affected  by  this  selection  bias;  this  is  a  common  analytical  challenge  of  working 
with  survey  data. 

'  Thus,  our  analysis  of  the  correlation  between  current  survey  teachers’  pedagogies  and 
previous  cadets’  outcomes  has  an  underlying  assumption  that  the  distribution  of  teachers’ 
characteristics  and  teaching  approaches  is  constant  over  time. 
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regarding  postresidential  outcomes  will  therefore  be  least  accurate  for  programs 
with  significant  teacher  turnover  since  the  postresidential  cadets  in  the  2014  reports 
graduated.  To  gauge  how  much  of  a  problem  this  is,  we  asked  each  ChalleNGe 
program  to  provide  information  on  their  annual  teacher  turnover  in  recent  years  (for 
as  far  back  as  they  have  data).  The  average  teacher  turnover  rate  is  22  percent, 
meaning  that  roughly  one  in  five  teachers  turns  over  in  a  given  year.®  Thus,  on 
average,  78  percent  of  the  teachers  who  completed  our  survey  should  be  the  same 
teachers  who  instructed  those  cadets  whose  postresidential  information  is  included 
in  the  2014  annual  reports.  The  minimum  annual  turnover  rate  is  0  percent  in  Idaho 
(although  this  is  based  on  only  two  years  of  data),  and  the  maximum  is  44  percent  in 
New  Mexico  (based  on  15  years  of  available  data). 


®  The  ChalleNGe  programs  provided  us  with  the  turnover  information  they  had  available  and, 
as  a  result,  the  program-level  turnover  calculations  were  based  on  different  numbers  of  years 
of  data.  On  average,  programs  had  seven  years  of  teacher  turnover  data  available,  with  a 
minimum  of  two  years  and  a  maximum  of  18  years. 
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In  this  section,  we  provide  summary  statistics  on  our  survey  population  and 
highlight  how  they  vary  across  ChalleNGe  programs.  This  information  provides 
background  for  our  results  and  characterizes  our  population  (and  how  it  might  differ 
from  the  ChalleNGe  teacher  population  at  large)  so  that  we  can  identify  ways  in 
which  our  findings  might  be  biased.  We  begin  by  discussing  our  survey  response 
rates  and  then  summarize  the  characteristics  of  our  sample’s  teachers— including  the 
type  of  program  at  which  they  teach,  the  subjects  they  teach,  their  educational 
background,  their  teaching  experience,  and  whether  they  hold  a  teaching  certificate. 

Response  rates 

Our  overall  response  rate  was  72  percent:  we  sent  the  survey  to  252  teachers  and 
received  181  replies.^  Figure  1  illustrates  the  response  rates  for  each  of  the  35 
ChalleNGe  programs.  More  than  a  third  of  the  programs  had  100-percent  response 
rates.  At  the  other  extreme,  one  program  returned  no  surveys:  Louisiana-Camp 
Beauregard.  Other  programs  with  noticeably  low  response  rates  include  West  Virginia 
(25  percent).  South  Carolina  (25  percent);  Louisiana-Camp  Minden  (36  percent),  and 
Florida  (43  percent).  All  other  programs  had  response  rates  of  50  percent  or  better. 
On  the  whole,  these  are  remarkably  good  response  rates:  at  least  half  the  contacted 
teachers  responded  in  30  of  the  35  programs. 

Given  these  response  rates,  we  have  essentially  oversampled  from  the  high-response- 
rate  programs  and  undersampled  from  the  low-response-rate  programs.  This  has  the 
effect  of  overweighting  teachers’  experiences  in  the  former  and  underweighting  those 
in  the  latter.  If  the  programs  with  noticeably  low  or  high  response  rates  are 
systematically  different  from  those  programs  whose  response  rates  are  closer  to  the 
average,  our  findings  may  not  be  representative  of  the  ChalleNGe  teacher  population 


^  Some  programs  returned  more  surveys  than  we  originally  sent.  This  happened  in  cases  where 
paper  surveys  were  mailed,  and  some  teachers  took  it  upon  themselves  to  photocopy  and 
distribute  the  survey  to  other  teachers  who  had  not  received  it.  For  the  purpose  of  calculating 
response  rates,  we  do  not  include  these  “additional”  surveys  in  our  calculations.  Thus,  the 
response  rate  is  capped  at  100  percent. 
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as  a  whole. It  is  important  to  keep  these  potential  discrepancies  in  mind  in 
interpreting  our  results. 


Figure  1.  Survey  response  rates,  by  ChalleNGe  program^ 


16 

14 


12  ^ 
(U 

10  I 


Q) 

E 

3 


Source:  CNA  tabulations  of  teacher  survey  responses. 

The  black  line  indicates  the  numberof  suveyssentto  each  ChalleNGe  program  (and 
thus  the  numberincluded  in  ourcalculation  of  the  response  rate). 


Program  type 

Depending  on  the  program  they  attend,  how  many  high  school  credits  they  have 
acquired,  and  their  age,  ChalleNGe  cadets  are  working  toward  one  of  three  goals. 
Cadets  who  successfully  complete  ChalleNGe  leave  the  program  with  a  high  school 
(HS)  diploma,  a  GED  certificate,  or  credits  enabling  them  to  return  to  their  home  high 


One  particular  difference  for  which  we  tested  is  whether  there  is  a  significant  correlation 
between  response  rate  and  program  size  (as  proxied  by  the  number  of  teachers).  We  found  no 
statistically  significant  relationship  between  these  two  factors:  teachers  from  larger  programs 
were  no  more  or  less  likely  to  respond  than  those  from  smaller  programs.  In  addition,  we 
investigated  whether  response  rates  differed  systematically  with  our  outcome  variables  of 
interest  (e.g.,  graduation  rate,  placement  rates);  they  do  not. 
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schools  and  ultimately  graduate  (this  process  is  called  credit  recovery).  In  Figure  2, 
we  display  the  number  of  teachers  affiliated  with  each  program  type,  as  well  as  with 
programs  offering  more  than  one  option  to  their  cadets.  The  majority  of  our  survey 
participants  (63  percent)  come  from  either  a  GED-only  program  or  a  program 
offering  all  three  options.  Only  2  percent  of  the  teachers  in  our  sample  are  from 
credit-recovery-only  programs,  and  only  6  percent  are  from  HS-diploma-only 
programs.  Most  are  from  programs  that  offer  some  combination  of  options. 

The  corresponding  distribution  of  teachers  for  the  ChalleNGe  population  as  a  whole  is 
presented  in  Figure  3.  The  two  program  types  in  which  our  survey  distribution  differs 
most  from  the  distribution  of  all  ChalleNGe  teachers  are  shown  in  green  (GED  and  HS 
diploma)  and  light  blue  (GED,  HS  diploma,  and  credit  recovery);  even  in  these  cases, 
the  differences  are  not  great.  In  the  ChalleNGe  program,  teachers  in  GED  and  HS 
diploma  (green)  programs  make  up  20  percent  of  all  teachers,  versus  17  percent  of 
our  sample;  those  who  teach  in  GED,  HS-diploma,  and  credit-recovery  (light  blue) 
programs  make  up  26  percent  of  all  ChalleNGe  teachers,  versus  31  percent  of  our 
teachers.  Thus,  we  are  somewhat  overrepresented  in  the  latter  category  and  somewhat 
underrepresented  in  the  former.  Given  that  the  differences  are  only  3  and  5 
percentage  points,  we  do  not  expect  that  this  will  introduce  any  bias  into  our  results. 


Figure  2.  Distribution  of  teachers'  program  types(GED  granting,  HSdiploma 
granting,  credit  recovery) 


■  GED  only 

■  HS  diploma  only 

■  Credit  recovery  only 

■  GED  and  credit  recovery 

■  GED  and  HS  diploma 

■  HS  diploma  and  credit  recovery 

■  GED,  HS  diploma,  and  credit 
recovery 


Source:  CNA  tabulationsof  tea  c  her  survey  data. 
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Figure  3.  Program-wide  distribution  of  teachers,  by  program  type 


■  GED  only 

■  HS  diploma  only 

■  Credit  recovery  only 

■  GED  and  credit  recovery 

■  GED  and  HS  diploma 

■  HS  diploma  and  credit 
recovery 

■  GED,  HS  diploma,  and 
credit  recovery 


Source:  CNA  tabulations  of  annual  report  data. 


Subjects  currently  taught 

Figure  4  and  Table  1  provide  information  on  the  subjects  our  survey  participants 
teach  at  ChalleNGe,  both  nationally  and  at  the  program  level.  As  Figure  4  shows,  126 
of  the  191  teachers  we  surveyed  indicated  that  they  teach  English/language  arts, 
followed  in  frequency  by  math,  science,  social  studies,  and  “other.” 


Figure  4.  Distribution  of  subjects  taught  (all  ChalleNGe  programs)^ 


0%  20%  40%  60%  80%  100% 

■  Math  ■  Science  ■  Social  Studies  ■  EngLangArts  ■  Other 


Source:  CNA  tabulationsof  tea  c  her  survey  data. 

a-  The  numbers  within  the  barsum  to  more  than  191  (the  total  numberof  teachers 
surveyed)  because  many  ChalleNGe  teachers  teach  more  than  one  subject. 
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Table  1  illustrates  the  same  information  at  the  program  level.  Although  most 
programs  largely  follow  this  pattern,  some  differences  exist  across  the  programs. 


Table  1  Distribution  of  subjec ts currently  taught,  byChalleNGe  program 


Program 

Math 

Science 

Social 

Studies 

English/ 

Language 

Arts 

Other 

AK 

4 

4 

4 

4 

4 

AR 

2 

-- 

2 

2 

-- 

CA-G  rizzly 

6 

4 

4 

10 

8 

CA-Sunburst 

4 

4 

4 

6 

-- 

DC 

4 

4 

-- 

2 

-- 

FL 

-- 

-- 

-- 

2 

2 

GA-Fort  Gordon 

4 

4 

2 

4 

-- 

GA-Fort  Stewart 

4 

4 

6 

6 

2 

Hl-Hilo 

4 

-- 

-- 

2 

-- 

Hl-Kapolei 

-- 

-- 

-- 

-- 

6 

ID 

2 

2 

-- 

2 

4 

IL 

4 

10 

8 

8 

4 

IN 

-- 

-- 

-- 

2 

2 

KY-Appalachian 

6 

6 

6 

6 

2 

KY-BluegrBSS 

4 

4 

4 

4 

2 

LA-Cannp  Minden 

6 

6 

6 

6 

6 

LA-GillisLong 

16 

8 

8 

14 

8 

MD 

-- 

-- 

2 

2 

-- 

Ml 

2 

2 

2 

2 

2 

MS 

8 

8 

6 

6 

-- 

MT 

-- 

2 

2 

2 

-- 

NC 

2 

2 

2 

2 

4 

NJ 

-- 

2 

2 

2 

-- 

NM 

4 

4 

2 

2 

-- 

OK 

2 

-- 

4 

4 

6 

OR 

10 

4 

-- 

8 

6 

PR 

2 

4 

2 

2 

12 

SC 

2 

2 

-- 

-- 

-- 

IX 

6 

2 

2 

4 

2 

VA 

4 

4 

4 

4 

2 

WA 

2 

2 

-- 

-- 

-- 

Wl 

2 

2 

2 

2 

WV 

-- 

-- 

-- 

-- 

2 

WY 

2 

2 

2 

2 

-- 

Source:  CNA  tabulationsof  teachersutvey  data. 
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Educational  qualifications 

Teachers’  educational  backgrounds  vary  significantly  across  programs,  as  shown  in 
Figure  5.  The  survey  asked  teachers  what  credentials  or  degrees  they  held.  Using  the 
highest  degree  or  credential  reported  by  each  teacher,  we  found  that  124  ChalleNGe 
teachers  have  bachelor’s  degrees,  72  have  master’s  degrees,  15  are  “education 
specialists”  (requiring  additional  coursework  beyond  a  master’s),  2  have  doctorates 
in  education,  and  3  indicated  that  they  have  “other  professional  degrees.”  Due  to  the 
predominance  of  bachelor’s  and  master’s  degrees,  we  decided  to  separate  the 
teachers  into  two  categories:  those  having  “more  than  a  bachelor’s”  (45  percent  of 
our  entire  sample,  most  of  whom  hold  a  master’s  degree)  and  those  with  a 
“bachelor’s  degree.” 

Figure  5.  Percentage  of  teachers  holding  more  than  a  bacheloFsdegree,  by 

ChalleNGe  progrann^'b 

100% 

90% 

80% 

70% 

60% 

50% 

40% 

30% 

20% 

10% 

0% 


30 


25 


20 
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Q. 


10 


Oj 

E 

3 


Source:  CNA  tabulationsof  tea  c  her  survey  data. 

3-  The  red  line  in  the  figure  represents  the  sample  average.  The  black  line  correspondsto 
the  secondary  vertical  axisand  represents  the  numberof  responses  included  in  each  blue 
bar^spercentage  calculation. 

b-  Forthe  programs  with  no  blue  bar,  no  teachers  indicated  thatthey  held  more  than  a 
bachelor^sdegree  (including  those  who  did  not  reply  to  this  question). 


There  is  significant  variation  in  the  prevalence  of  teachers  holding  more  than  a 
bachelor’s  degree  across  programs.  In  fact,  10  of  the  34  programs  for  which  we  have 
survey  data  have  no  teachers  with  education  beyond  a  bachelor’s  degree.  An 
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additional  10  programs  have  a  smaller  percentage  of  teachers  with  bachelor’s 
degrees  on  their  staff  than  the  ChalleNGe  national  average.  The  remaining  14 
programs  are  above  average  in  this  measure,  the  most  notable  being  Florida,  Georgia- 
Fort  Stewart,  and  West  Virginia:  100  percent  of  their  teachers  in  our  survey 
population  have  more  than  a  bachelor’s  degree. 

Other  qualification  measures 

The  last  three  “qualification”  measures  we  compare  across  programs  are  the 
teachers’  average  years  of  experience  (overall  and  at  ChalleNGe)  and  the  percentage 
of  teachers  holding  a  teaching  certificate.  Figure  6  and  Figure  7  display  the  two 
measures  of  teaching  experience.  The  average  years  of  teaching  experience  for  the 
entire  sample,  as  shown  by  the  red  line  in  Figure  6,  is  14.  As  the  figure  illustrates, 
teachers  in  a  handful  of  programs  average  more  than  20  years  of  experience  (Florida, 
Louisiana-Gilhs  Long,  New  Mexico,  and  West  Virginia);  these  programs  are  increasing 
the  sample  average.  This  variation  likely  exists  because  teachers  arrive  at  ChalleNGe 
at  different  stages  of  their  careers:  some  will  likely  retire  after  leaving  ChalleNGe, 
and  others  teach  at  ChalleNGe  at  the  begirming  of  their  careers.  There  is  noticeably 
less  variation  across  programs  in  the  teachers’  average  years  teaching  ChalleNGe,  as 
shown  in  Figure  7  (note  that  the  y-axis  scales  are  different  in  the  two  figures).  The 
programs  that  stand  out  in  that  figure  are  Alaska,  with  an  average  ChalleNGe-specific 
experience  of  12  years,  as  well  as  those  whose  teachers  on  average  have  fewer  than  2 
years  of  ChalleNGe  experience:  Hawaii-Kapolei,  Idaho,  Kentucky-Appalachian,  and 
West  Virginia. 

As  shown  in  Figure  8,  the  percentage  of  teachers  holding  teaching  certificates  is  high, 
both  across  the  sample  as  a  whole  and  within  most  ChalleNGe  programs.  The 
national  average  for  survey  respondents  (shown  by  the  red  hue)  is  78  percent,  and 
the  program-specific  average  is  100  percent  for  18  of  the  34  programs  whose 
teachers  participated.  Those  with  particularly  low  teacher  certification  rates  include 
the  District  of  Columbia  (at  20  percent),  both  of  the  Georgia  programs  (at  33 
percent),  and  Maryland  (at  33  percent).  Four  programs’  survey  responses  indicated  a 
0-percent  teacher  certification  rate:  Alaska,  Hawaii-Kapolei,  Kentucky-Bluegrass,  and 
New  Mexico.”  It  is  not  necessarily  the  case  that  teachers  without  teaching  certificates 
do  not  meet  their  states’  teaching  requirements,  as  they  may  still  have  teaching 
licenses.  It  is  clear  from  Figure  8  that  at  least  some  survey  respondents  differentiated 
between  licenses  and  certificates. 


“  It  is  important  to  note  that  the  statistics  reported  here  and  throughout  this  report  are  based 
on  survey  respondents  only.  In  this  case,  if  any  certified  teachers  did  not  respond  to  our  survey 
or  responded  but  did  not  answer  the  question  on  qualifications,  our  estimates  of  teacher 
certification  rates  will  be  lower  than  the  tme  rates. 
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Figure  6.  Average  years  of  teaching  experience,  by  ChalleNGe  progrann^ 
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Source:  CNA  tabulationsof  tea  c  her  survey  data. 

3-  The  red  line  in  the  figure  represents  the  sample  average.  The  black  line  correspondsto 
the  secondary  vertical  axisand  represents  the  numberof  responses  included  in  each  bar^s 
calculation. 
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Figure  7.  Average  yearsof  ChalleNGe  teaching  experience,  by  ChalleNGe 
program^ 


Source:  CNA  tabulationsof  tea  c  her  survey  data. 

3-  The  red  line  in  the  figure  represents  the  sample  average.  The  black  line  correspondsto 
the  secondary  vertical  axisand  represents  the  numberof  responses  included  in  each  bar^s 
calculation. 
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Percentage  of  teachers  with  a  teaching  certificate,  by  ChalleNGe 

program^-b 


Source:  CNA  tabulationsof  tea  c  her  survey  data. 

3-  The  red  line  in  the  figure  represents  the  sample  average.  The  black  line  correspondsto 
the  secondary  vertical  axisand  represents  the  numberof  responses  included  in  each  bar^s 
calculation. 

b-  Programs  with  no  blue  barhad  no  teachers  with  a  teaching  certificate  among  their 
survey  respondents. 


Having  characterized  our  survey  population  in  terms  of  response  rates,  program 
type,  subjects  taught,  and  basic  teacher  qualifications,  we  now  turn  to  regression 
analysis  of  how  differences  in  teacher  qualifications  affect  cadets’  outcomes.  The 
following  sections  address  pedagogical  differences  and  teachers’  best  practices. 


17 


Teachers’  Qualifications 


In  this  section,  we  investigate  whether  a  program’s  average  teacher  qualifications  are 
correlated  with  its  graduation  rate,  TABE  score  improvements,  and  postresidential 
placement  rates.  In  addition  to  the  measures  of  teachers’  qualifications  introduced  in 
the  previous  section  (highest  degree  held,  years  of  teaching  experience,  and  teaching 
certification),  we  also  consider  the  importance  of  teachers’  majors  and  coursework— 
specifically,  if  the  teaching  staff  majored  or  took  coursework  in  the  subject(s)  they 
teach  at  ChalleNGe. 

Table  2  presents  the  distribution  of  college  (or  postgraduate)  “majors”  by  the  subject 
taught  at  ChalleNGe.  The  percentages  should  be  read  by  colunms  (not  across  rows). 
Among  teachers  who  teach  science  at  ChalleNGe,  for  example,  the  percentages  in  that 
colunrn  indicate  that  38  percent  of  those  teachers  majored  in  science,  40  percent 
majored  in  education,  15  percent  majored  in  the  social  sciences,  and  10  percent 
majored  in  social  studies.  We  include  education  and  social  sciences  among  our  list  of 
majors,  in  part  due  to  their  high  prevalence,  even  though  these  are  not  subjects 
explicitly  taught  at  ChalleNGe.  Cadets  do  not,  for  example,  take  classes  in  economics, 
sociology,  or  any  other  social  sciences,  nor  do  they  take  classes  in  education.  Later  in 
this  section,  when  we  measure  the  percentage  of  teachers,  by  program,  who  majored 
in  the  subject  matter  they  teach  at  ChalleNGe,  those  with  majors  in  social  sciences  or 
education  will  be  counted  among  those  who  did  not  major  in  what  they  teach  at 
ChalleNGe. 

A  few  patterns  in  this  table  are  worth  noting.  The  most  common  major,  regardless  of 
the  subject  taught  at  ChalleNGe,  is  education.  We  suspect  that  this  is  preferable— in 
terms  of  leading  to  better  outcomes  for  students— to  mastery  of  the  subject  matter 
taught.  Specifically,  we  expect  that  understanding  how  to  cormect  with  students  and 
effectively  vary  pedagogical  methods— especially  given  the  ChalleNGe  population— is 
more  important  in  realizing  positive  cadet  outcomes  than  a  teacher’s  subject-specific 
knowledge  base.  We  will  test  this  later  in  this  section,  when  we  use  teacher 
qualifications  as  predictors  of  cadets’  outcome  measures.  Among  science. 
English/language  arts,  and  social  studies  teachers,  the  second  most  common  major  is 
the  subject  they  teach.  This  is  not  the  case,  however,  for  math  teachers.  In  fact,  math 
is  among  the  least  common  majors  for  math  teachers.  Given  the  level  of  math  being 
taught  in  these  classrooms,  this  is  likely  not  a  cause  for  concern. 
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Table  2.  ChalleNGe  teachers'  majors,  by  subject  taught^ 


Major 

Math 

For  ChalleNGe  teachers  who  teach: 

English/ 

Language  Social 

Science  Arts  Studies 

Other 

Math 

9% 

4% 

0% 

0% 

0% 

Science 

13% 

38% 

5% 

4% 

14% 

English/ Language  Arts 

4% 

4% 

25% 

12% 

3% 

Social  Studies 

13% 

10% 

14% 

24% 

14% 

Social  Sciences 

13% 

15% 

11% 

17% 

9% 

Art 

7% 

4% 

4% 

5% 

14% 

Education 

51% 

40% 

48% 

44% 

69% 

Other 

5% 

4% 

5% 

2% 

0% 

Observations 

55 

48 

56 

41 

35 

Source:  CNA  tabulationsof  teachersurvey  data. 

®1he  column  percentagesdo  notadd  to  100  because  many  teachers  have  more  than 
one  major.  That  is,  the  majors  in  this  table  are  not  mutually  exclusive. 

Having  synthesized  the  prevalence  of  different  major  types  across  ChalleNGe 
teachers,  by  subject  taught,  for  the  sample  as  a  whole,  we  now  turn  to  an  analysis  of 
how  teachers’  quahfications  in  terms  of  their  majors  (and  coursework)  vary  across 
programs.  We  present  the  distribution  of  five  “qualification  categories”  in  Figure  9. 
We  have  created  these  variables  to  be  mutually  exclusive  in  the  order  of  relevance  to 
what  they  currently  teach.  Thus,  we  begin  by  creating  a  variable  that  takes  a  value  of 
1  if  teachers  majored  in  the  subject  they  teach  at  ChalleNGe,  and  0  otherwise.  We 
then  create  a  variable  that  indicates  the  number  of  teachers  per  program  who 
majored  in  education,  but  this  includes  only  those  teachers  who  did  not  major  in 
what  they  teach  at  ChalleNGe.  The  third  variable  captures  the  number  of  teachers 
with  coursework  in  the  subject  they  teach  at  ChalleNGe  (given  that  they  majored 
neither  in  this  subject  nor  in  education),  and  the  fourth  captures  the  number  with 
coursework  in  education  (given  that  the  three  previous  measures  were  false).  Finally, 
the  fifth  variable  captures  the  number  of  teachers  per  program  who  meet  none  of 
these  quahfications. 

These  percentages  are  illustrated  in  Figure  9  and  are  program-specific.  That  is,  they 
illustrate  the  percentages  of  teachers,  within  a  program,  who  fall  into  each  of  the  five 
categories.  The  five  categories  are  mutually  exclusive  in  the  following  order:  major  in 
subject  taught,  major  in  education,  coursework  in  subject  taught,  coursework  in 
education,  and  none  of  the  above.  The  “major  in  education”  variable,  for  example, 
takes  a  value  of  1  only  for  teachers  who  majored  in  education  but  did  not  major  in 
the  subject  they  teach  at  ChalleNGe.  As  Figure  9  illustrates,  teacher  quahfication  by 
major  varies  significantly  across  the  ChalleNGe  programs.  The  programs  with  the 
most  qualified  teachers  (by  these  measures)  are  Georgia-Fort  Stewart,  Indiana,  and 
Idaho— the  only  three  programs  where  less  than  40  percent  of  the  teachers  meet 
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none  of  the  major  or  coursework  qualifications.  At  the  other  extreme  are  Alaska  and 
Hawaii-Hilo,  with  89  and  86  percent  of  teachers,  respectively,  meeting  none  of  these 
qualifications.  We  find  it  particularly  striking  that  at  all  programs,  at  least  30  percent 
of  the  teachers  participating  in  our  survey  meet  none  of  the  qualifications  displayed 
in  Figure  9;  this  percentage  seems  high  given  that  the  lowest  level  of  qualification  we 
use  is  only  having  taken  coursework  in  education  (a  low  standard  to  meet). 


Figure  9.  ChalleNGe  teachers'  majorsand  coursework  (and  their  relevance  to 
what  they  teach),  by  ChalleNGe  program 


U^Q:a:ux<<>^> 


■  Major  in  subject  taught  ■  Major  in  education 

■  Coursework  in  subject  taught  ■  Coursework  in  education 

■  None  of  these 


Source:  CNA  tabulationsof  tea  c  her  survey  data. 


In  the  remainder  of  this  section,  we  evaluate  whether  the  programs’  average  teacher 
qualification  rates  (by  various  measures)  are  correlated  with  the  programs’  average 
cadet  outcomes.  We  consider  five  cadet  outcome  variables:  program  graduation  rate, 
TABE  improvement  (pre  to  post),  1 -month  placement  rate,  6-month  placement  rate, 
and  12-month  placement  rate.  We  determine  whether  each  outcome  is  correlated 
with  the  programs’  average  teacher  experience  (both  overall  and  ChalleNGe-specific), 
the  percentage  of  teachers  holding  a  teaching  certificate,  the  percentage  with  more 
than  a  bachelor’s  degree,  and  the  percentage  whose  majors  are  relevant  to  what  they 
teach  (either  because  they’re  teaching  the  subject  in  which  they  majored  or  because 
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they  majored  in  education).  The  relationships  between  all  these  teacher 
characteristics  with  our  outcome  variables  of  interest  are  displayed  in  Table  3. 

Table  3.  Marginal  changes  in  average  cadet  outcomes  associated  with  changes 
in  average  teacherqualifications^'^ 


Program 

graduation 

rate 

TABE 

improve¬ 

ment 

Placement  rate 

1 -month 

6-month 

12-month 

Average  teachers' 
experience  (in  years) 

Insig. 

-0.04* 

Insig. 

Insig. 

Insig. 

Average  teachers' 
ChalleNGe 
experience  (in  years) 

Insig. 

Insig. 

Insig. 

Insig. 

Insig. 

Percentage  with 
teaching  certificate 

Insig. 

Insig. 

Insig. 

Insig. 

Insig. 

Percentage  with  more 
than  a  bachelor's 
degree 

Insig. 

0.88* 

0.32** 

0.27** 

0.22* 

Percentage  that 
majored  in  what  they 
teach 

Insig. 

-1.52** 

Insig. 

Insig. 

Insig. 

Percentage  that 
majored  in  education 

Insig. 

Insig. 

Insig. 

Insig. 

Insig. 

Numberof 

observations 

34 

34 

33 

33 

33 

Source:  CNA  estimations  using  teacher  survey  data. 

3-  The  marginal  changesdisplayed  in  this  table  represent  the  change  in  cadets'  average 
graduation  rates,  TABE  improvement,  and  postresidential  placement  ratesthat  are 
associated  with  a  one-unit  increase  in  the  teacherqualifications(eitheran  increase  of  1 
yearin  experience  ora  1-percentage-point  increase  in  the  othervariables). 

"Insig."  indicates  that  the  relationship  between  that  teacher  qualification  and  the 
outcome  variable  of  interest  is  statistic  ally  insignificant. 

*  Indicates  that  the  relationship  isstatistically  significant  at  the  10-percent  level.  That  is,  the 
result  would  occur  by  chance  fewer  than  10  times  in  100. 

**  Indicates  that  the  relationship  isstatistically  significant  at  the  5-percent  level.- 


A  few  noteworthy  findings  emerge  from  the  findings.  First,  none  of  the  teacher 
qualification  variables  is  a  significant  predictor  of  program  graduation  rates.  It  is 
important  to  recall  that  these  are  program-level,  average  regressions  and  that 
individual  cadets  are  not  being  matched  to  specific  teachers.  We  suspect  that  more 
statistically  significant  relationships  would  emerge  if  we  were  able  to  match  cadets 
and  teachers  in  this  way.  Second,  the  percentage  of  teachers  at  a  program  with  more 
than  a  bachelor’s  degree  has  a  significant  relationship  with  four  of  our  five  outcome 
measures  (all  but  graduation  rate).  This  suggests  that  a  greater  prevalence  of  more 
highly  educated  teachers  has  positive  impacts  on  overall  cadet  performance.  These 
relationships  are  also  sizable— a  1 -percentage-point  increase  in  the  percentage  of  a 
program’s  teachers  who  hold  more  than  a  bachelor’s  degree  is  associated  with  a 
32-percent  increase  in  the  1-month  placement  rate,  a  27-percent  increase  in  the  6- 
month  placement  rate,  a  22-percent  increase  in  the  12-month  placement  rate,  and  an 
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increase  of  0.86  in  the  average  TABE  improvement.^^  Finally,  we  also  find  that 
teachers’  average  teaching  experience  is  negatively  related  to  the  cadets’  average 
TABE  improvement,  as  is  the  percentage  who  majored  in  what  they  teach.  This  may 
indicate  that  older  teachers  (who  have  more  years  of  teaching  experience)  and  those 
with  degrees  in  what  they  teach  are  either  less  focused  on  TABE  improvement  or  are 
less  familiar  with  the  methods  that  most  effectively  improve  cadets’  TABE 
performance.  But,  again,  these  suppositions  cannot  be  confirmed  without  individual- 
level  teacher  and  cadet  data. 

Overall,  we  have  found  that  few  of  the  teacher-qualification  characteristics  (in  the 
aggregate)  have  significant  relationships  with  cadets’  average  outcomes.  The  average 
ChalleNGe  teaching  experience  among  a  program’s  staff  has  no  significant  impact  on 
that  program’s  average  cadet  outcomes,  and  nor  does  the  percentage  of  teachers  who 
majored  in  the  subject  they  teach  at  ChalleNGe.  The  percentage  of  a  program’s 
teachers  who  majored  in  education  and  the  teachers’  average  overall  teaching 
experience  were  significantly  related  only  to  TABE  improvement— and  negatively  so. 
Our  results  suggest  that  the  most  important  teacher  qualification  is  the  percentage 
of  teachers  who  have  attained  more  than  a  bachelor’s  degree.  This  was  positively  and 
significantly  associated  with  all  cadet  outcomes,  except  the  average  program 
graduation  rate,  which  we  do  not  find  to  be  significantly  associated  with  any  of  our 
measures  of  teachers’  qualifications.  These  results  suggest  that  programs  (and 
thereby  cadets)  might  well  gain  from  an  increased  percentage  of  teachers  with 
graduate  degrees.  We  recommend,  however,  that  individual-level  analysis  linking 
cadets  to  teachers  be  conducted  before  any  policy  implications  are  drawn  from  this 
report. 


Given  that  the  range  of  TABE  improvements  for  our  sample  (by  class  average)  is  from  0.1  to 
4.3,  an  increase  of  0.88  represents  20  percent  of  the  total  4.3  point  range. 
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In  this  section,  we  review  teachers’  inputs  regarding  their  pedagogical  approaches 
and  investigate  whether  any  program-level  differences  are  correlated  with  program- 
level  outcomes  for  cadets  (e.g.,  program  graduation  rate,  TABE  score  changes).  We 
first  present  our  analysis  of  the  more  general  pedagogical  questions— asked  of  all 
teachers— and  have  grouped  these  into  three  categories:  how  cadets  are  assigned 
both  to  classes  and  to  groups  within  classes;  how  teachers  use  their  instructional 
time;  and  overall  pedagogical  methods.  We  then  review  teachers’  responses  to 
subject-specific  survey  questions,  asked  only  of  teachers  of  math  or 
English/language  arts.  Since  competency  exams  (such  as  the  GED)  focus  on  these 
subject  areas,  we  found  it  pertinent  to  gather  additional  information  on  these 
teachers’  pedagogical  approaches. 

Cadet  class  assignments 

We  asked  teachers  how  cadets  at  their  program  are  assigned  to  specific  classes,  and 
provided  the  options  of  ability  or  previous  achievement,  age,  gender,  and  “other.”  A 
sufficient  number  of  teachers  selected  “other,”  explaining  that  students  were 
assigned  to  classes  by  platoon  or  alphabetically,  that  we  coded  these  as  separate 
categories.  Table  4  displays  the  distribution  of  responses  we  received,  by  ChalleNGe 
program.  Across  the  sample  as  a  whole,  57  percent  of  teachers  indicated  that  cadets 
are  assigned  to  specific  classes  based  on  ability  or  previous  performance,  17  percent 
based  on  gender,  3  percent  based  on  age,  5  percent  alphabetically,  9  percent  by 
platoon,  and  11  percent  said  “other”. These  numbers  sum  to  more  than  100 
because  teachers  were  asked  to  check  all  that  apply.  These  general  trends  are  largely 
mirrored  in  Table  4,  where  the  largest  concentration  of  responses  is  in  the  “ability” 
column,  followed  by  gender,  other,  platoon,  age,  and  alphabetical.  At  a  number  of 
programs,  100  percent  of  participating  teachers  indicated  that  assignments  are  made 
based  on  ability,  and  in  a  few  cases  0  percent  chose  ability  (these  programs  include 
Georgia-Eort  Gordon,  Illinois,  Mississippi,  New  Mexico,  Washington,  Wisconsin,  and 
Wyoming).  Other  noticeable  outliers  include  programs  assigning  cadets  to  classes 
alphabetically,  such  as  Michigan,  Puerto  Rico,  and  Wyoming. 


We  note  that  the  sample  average  of  17  percent  for  gender-based  class  assignments  is 
inconsistent  with  anecdotal  evidence  (from  discussions)  with  a  number  of  program  directors. 
That  is,  directors  indicated  that  class  assignments  are  made  primarily  based  on  gender,  since 
mixed-gender  classes  were  too  disruptive.  The  fact  that  only  17  percent  of  teachers  indicated 
that  class  assignments  are  gender-based  at  their  programs  suggests  that  they  may  not  have 
understood  the  nature  of  this  question. 


23 


CNA 

Y^i*^  a  ^rw  t  jTinN^? 


Table  4.  Distribution  of  class  assignment  methods,  by  ChalleNGe  program 


Program 

Ability 

Gender 

Age 

Alphabetical 

Platoon 

Other 

AK 

75% 

0% 

0% 

0% 

0% 

25% 

AR 

100% 

0% 

0% 

0% 

0% 

0% 

CA-G  rizzly 

100% 

0% 

0% 

0% 

0% 

0% 

CA-Sunburst 

91% 

9% 

0% 

0% 

0% 

0% 

DC 

80% 

20% 

0% 

0% 

0% 

0% 

FL 

67% 

0% 

0% 

0% 

0% 

33% 

GA-Fort  Gordon 

0% 

57% 

0% 

0% 

29% 

29% 

GA-Fort  Stewart 

33% 

0% 

33% 

0% 

0% 

33% 

Hl-Hilo 

50% 

50% 

0% 

0% 

0% 

0% 

Hl-Kapolei 

33% 

67% 

0% 

0% 

0% 

0% 

ID 

100% 

0% 

0% 

0% 

0% 

0% 

IL 

0% 

13% 

0% 

0% 

75% 

13% 

IN 

100% 

0% 

0% 

0% 

0% 

0% 

KY-Appalachian 

50% 

33% 

0% 

0% 

17% 

0% 

KY-Bluegrass 

100% 

0% 

0% 

0% 

0% 

0% 

LA-Camp  Minden 

80% 

0% 

0% 

0% 

20% 

20% 

LA-GillisLong 

85% 

0% 

0% 

0% 

0% 

15% 

MD 

33% 

67% 

0% 

0% 

0% 

0% 

Ml 

33% 

50% 

0% 

17% 

0% 

17% 

MS 

0% 

67% 

17% 

0% 

17% 

0% 

MT 

100% 

0% 

0% 

0% 

0% 

0% 

NJ 

50% 

50% 

0% 

0% 

0% 

0% 

NM 

0% 

50% 

0% 

0% 

0% 

50% 

NC 

38% 

25% 

0% 

0% 

25% 

13% 

OK 

100% 

0% 

0% 

0% 

0% 

0% 

OR 

100% 

0% 

0% 

0% 

0% 

0% 

PR 

11% 

0% 

22% 

56% 

0% 

11% 

SC 

100% 

0% 

0% 

0% 

0% 

0% 

IX 

63% 

0% 

13% 

0% 

0% 

25% 

VA 

71% 

0% 

0% 

0% 

0% 

29% 

WA 

0% 

25% 

25% 

0% 

25% 

25% 

WV 

100% 

0% 

0% 

0% 

0% 

0% 

Wl 

0% 

60% 

0% 

0% 

60% 

0% 

WY 

0% 

25% 

0% 

75% 

0% 

25% 

Sample  average 

57% 

17% 

3% 

5% 

9% 

11% 

Source:  CNA  tabulationsof  teachersurvey  data. 
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To  determine  whether  cadets  were  advantaged  or  disadvantaged  by  the  class- 
assignment  mechanisms  used  at  their  programs,  we  investigated  whether  class- 
assignment  methods  correlate  significantly  with  our  program-level  outcome 
variables.  We  expected,  in  particular,  that  those  assigned  by  ability  might  have  better 
overall  outcomes  than  those  assigned  to  classes  by  other  methods.  We  did  not, 
however,  find  any  significant  relationship  between  any  class-assignment  method  and 
cadets’  average  outcomes.  It  is  important  to  recall  that  this  may  be  due  to 
insufficient  power  and  minimal  variation  in  our  dataset.  Specifically,  we  are  working 
with  only  33  observations  and  average  outcome  data;  therefore,  we  cannot  correlate 
the  outcomes  of  students  within  a  program  who  may  have  been  assigned  to  classes 
in  different  ways  with  their  individual-level  outcomes.  We  would  be  able  to  more 
accurately  answer  this  empirical  question,  along  with  the  others  in  this  section,  if  we 
were  working  with  individual-level  (instead  of  program-level)  data. 

Our  survey  also  asked  if  teachers  create  groups  within  their  classes  based  on  ability 
or  previous  achievement.  The  majority  of  teachers  at  most  programs  indicated  that 
they  do  not  create  such  groups.  In  fact,  in  only  six  programs  did  the  majority  of 
teachers  indicate  that  they  do  create  ability-based  groups  within  their  classroom: 
Arkansas,  District  of  Columbia,  Georgia-Fort  Gordon,  Georgia-Fort  Stewart,  Indiana, 
and  Kentucky-Bluegrass  (see  Figure  10).  Once  again,  we  found  no  significant 
correlation  between  the  percentage  of  teachers  at  a  program  who  create  groups 
based  on  ability  and  the  average  outcomes  of  cadets.  As  previously  mentioned,  this 
may  be  because  of  the  insufficient  variation  and  power  in  our  data,  but  it  also  may  be 
because  those  who  indicated  that  they  do  create  groups  by  ability  do  so  in  different 
ways  (perhaps  some  grouping  those  of  similar  ability  and  others  intentionally  mixing 
ability  levels  within  groups).  If  this  is  the  case,  overall  effects  may  be  averaged  out. 
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Percentage  of  teachers  who  create  classgroupsbased  on  ability,  by 
ChalleNGe  program^b 
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Source:  CNA  tabulationsof  tea  c  her  survey  data. 

Teachers  at  prog  rams  with  no  visible  bar  indie  a  ted  that  they  do  not  create  groups  based 
on  ability,  with  the  exception  of  South  Carolina.  In  that  case,  no  teachers  answered  this 
question;  thus,  the  missing  barshould  be  interpreted  as  missing  data,  not  asa  “true  zero." 

The  red  line  in  the  figure  represents  the  sample  average.  The  black  line  correspondsto 
the  secondary  vertical  axisand  represents  the  numberof  responses  included  in  each  bar^s 
calculation. 


The  final  type  of  cadet  assignment  we  consider  is  whether  male  and  female  cadets 
take  classes  together  or  separately.  In  the  overall  sample,  11  percent  of  teachers 
indicated  that  male  and  female  cadets  take  their  class  together,  82  percent  indicated 
that  they  are  separate,  and  6  percent  said  it  varies.  The  program-specific  averages  for 
these  variables  are  presented  in  Figure  11.  Not  surprisingly,  100  percent  of  teachers 
at  most  programs  indicated  that  male  and  female  cadets  take  their  classes  separately 
since  this  is  likely  a  program-level  policy  in  most  cases  (with  little  variation  across 
classes  within  a  program).  Four  programs  stand  out  for  having  mixed-gender  classes 
among  100  percent  of  their  survey  respondents:  Alaska,  Montana,  Puerto  Rico,  and 
West  Virginia.  In  terms  of  correlation  between  mixed-  or  separate-gender  classes  and 
outcomes,  we  find  significance  only  for  the  12-month  placement  rate.  Specifically,  we 
find  that  programs  where  the  teachers  reported  that  male  and  female  cadets  are 
always  separate  or  always  together  have  higher  12-month  placement  rates  than  those 
reporting  that  it  varies.  This  suggests  that  gender  consistency  in  classroom 
composition  is  what  is  most  important  for  these  cadets  (whatever  the  policy). 
Perhaps,  for  example,  cadets  who  move  between  mixed-gender  and  gender-divided 
classes  are  distracted  by  the  change  in  classroom  environment,  or  perhaps 
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mixed-gender  classes  are  more  distracting  than  single-gender  ones.  Such 
suppositions  cannot  be  confirmed  without  more  detailed  analysis,  especially  since  we 
found  a  significant  correlation  only  with  the  12-month  placement  rate  and  not  with 
the  other  placement  rates,  the  program  graduation  rate,  or  the  TABE  improvement. 


Figure  11.  Percentage  of  teachers  who  indicated  that  male  and  female  cadets 
take  the  ire  lass  tog  ether,  separately,  or  both,  by  ChalleNGe  program^ 
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Source:  CNA  tabulationsof  tea  c  her  survey  data. 

The  blackline  in  the  figure  correspondsto  the  secondary  vertical  axisand  represents  the 
number  of  responses  included  in  each  bar^s  calculation. 


The  final  characteristic  we  consider  in  this  subsection,  because  it  is  likely  at  least 
partially  determined  by  class-assignment  practices,  is  average  class  size.  The  average 
class  size  reported  by  teachers  in  the  entire  sample,  as  illustrated  by  the  red  line  in 
Figure  12,  is  23  cadets.  Georgia-Fort  Gordon,  Oregon,  and  Puerto  Rico  stand  out  as 
having  average  class  sizes  much  larger  than  the  average  (all  above  30  cadets), 
whereas  the  District  of  Columbia,  Wyoming,  Kentucky-Bluegrass,  and  Mississippi  all 
have  noticeably  smaller  average  class  sizes  (all  less  than  15).  Since  class  size  is  likely 
directly  related  to  the  amount  of  individual  attention  teachers  can  provide  their 
students,  we  expected  to  find  statistically  significant  negative  correlations  between 
class  size  and  average  student  outcomes.  None  of  our  outcome  variables,  however,  is 
statistically  significantly  related  to  average  class  size.  Once  again,  we  cannot 
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determine  whether  this  is  because  class  size  has  no  impact  on  cadets’  performance 
and  the  overall  value  of  ChalleNGe  for  them,  or  if  our  average,  program-level  data 
have  insufficient  variation  to  capture  this  relationship. 


Source:  CNA  tabulationsof  tea  c  her  survey  data. 

^■The  South  Carolina  teachers  who  connpleted  the  survey  did  not  provide  this  information. 

b-  The  red  line  in  the  figure  represents  the  sample  average.  The  black  line  corresponds  to 
the  secondary  vertical  axisand  represents  the  numberof  responses  included  in  each  bar^s 
calculation. 


Division  of  teachers’  instructional  time 

We  asked  teachers  how  they  divide  their  instructional  time.  In  this  subsection,  we 
analyze  three  measures  of  teachers’  time:  (1)  the  amount  of  time  teachers  spend,  per 
week,  teaching  the  whole  class,  small  groups,  or  a  single  student;  (2)  hours  spent 
tutoring  per  week;  and  (3)  the  number  of  hours  spent  on  administrative  and 
disciplinary  tasks  (which  we  might  consider  “nonproductive”  hours  for  the  purposes 
of  cadets’  academic  progress). We  are  interested  not  only  in  how  these  measures 


There  is  potential  for  error  in  teachers’  responses  here  (and  in  response  to  any  other 
questions  regarding  how  they  divide  their  instructional  time).  This  is  because  teachers  were  not 
asked,  prior  to  the  survey,  to  keep  track  of  their  time,  but  rather  were  responding  to  these 
questions  based  on  their  best  recollections  about  time  that  had  already  passed.  That  said,  since 
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vary  across  programs,  but  also  in  whether  they  have  any  significant  relationships 
with  programs’  average  cadet-level  outcomes.  Figure  13  presents  teachers’  average 
number  of  hours  spent  per  week  on  whole-class,  small-group,  and  single -student 
instruction,  by  program.  The  survey  question  imposed  a  maximum  on  the  value  these 
variables  can  take.  Teachers’  time  choices  for  each  activity  were  none,  <1  hour,  1 
hour,  2  hours,  3  hours,  4  hours,  and  5+  hours.  There  were  a  total  of  seven  time-spent 
categories,  which  we  aggregated  into  whole-class,  small-group,  and  single-student 
instruction.^^  We  recoded  this  variable  to  take  values  of  0  through  5,  with  <1  hour 
and  1  hour  both  being  recoded  as  1.  Thus,  the  maximum  possible  value  for  any  bar 
in  Figure  13  is  35  (i.e.,  the  sum  of  hours  spent  on  the  three  aggregated  activity 
categories,  or  the  seven  separate  activities).^®  In  most  cases,  teachers  spend  the  most 
time  instructing  the  class  as  a  whole,  followed  by  small  groups,  and  then  instructing 
individual  students.  Although  the  amount  of  time  spent  on  these  different  types  of 
instruction  has  no  significant  correlation  with  either  cadets’  average  program 
graduation  rates  or  TABE  score  improvements,  we  do  find  one  significant 
relationship  with  a  postresidential  outcome.  Specifically,  our  OLS  regression  results 
indicate  that  programs’  average  1 -month  placement  rates  are  statistically 
significantly  negatively  correlated  with  the  number  of  hours  spent  instructing  the 
class  as  a  whole. This  suggests  that  students  benefit,  in  the  long  run,  from  more 
one-on-one  instruction  and  less  large-group  instruction. 


teachers  typically  plan  their  instructional  time  in  advance,  we  expect  them  to  have  more 
accurate  time-use  responses  than  might  be  true  of  the  general  population. 

Specifically,  whole-class  instruction  includes  all  hours  spent  “instructing  the  class  as  a 
whole”  and  “discussing  issues  as  a  class.”  Small-group  instruction  includes  hours  spent 
“instructing  small  groups  of  students”  and  “assisting  students  as  they  work  in  groups.”  Finally, 
single-student  instruction  includes  “instructing  single  students,”  “assisting  students  as  they 
work  independently,”  and  “assisting  students  as  they  work  on  homework.” 

Not  all  teachers’  time  with  students  can  necessarily  be  classified  as  one  of  our  seven 
subcategories.  In  addition,  teachers  may  classify  some  instructional  time  as  something  other 
than  the  categories  we  provided.  Thus,  some  of  the  exceptionally  low  numbers  displayed  in 
Figure  13  may  not  actually  reflect  the  average  hours  spent  instructing  students  at  those 
programs,  but  may  rather  reflect  only  the  average  hours  spent  on  a  portion  of  instruction  time. 

The  relationship  with  the  12 -month  placement  rate  is  significant  only  at  the  20-percent  level, 
which  is  outside  the  typically  reported  range  of  statistical  significance. 
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Figure  13.  Average  hoursspent  per  week  instructing  the  classasa  whole,  small 
groups  of  students,  and  single  students,  by  ChalleNGe  program^ 
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Source:  CNA  tabulationsof  tea  c  her  survey  data. 

The  black  line  in  the  figure  correspondsto  the  secondary  vertical  axisand  represents  the 
number  of  responses  included  in  each  bar's  calculation. 


Figure  14  displays  the  average  number  of  hours  spent  per  week  tutoring  or  working 
with  students  outside  class,  by  ChalleNGe  program.  The  sample  average,  shown  by 
the  red  line,  is  2  hours  per  week,  and  there  is  significant  variation  around  that 
average— with  program  averages  ranging  from  under  0.5  hour  to  over  4.5  hours. 
Although  we  expected  that  programs  with  higher  average  tutoring  hours  would  have 
higher  average  cadet  outcomes,  we  find  no  statistically  significant  relationship  for 
any  of  our  outcome  measures. 
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Source:  CNA  tabulationsof  tea  c  her  survey  data. 

The  red  line  represents  the  sannple  average.  The  black  line  correspondsto  the  secondary 
vertical  axisand  represents  the  numberof  responses  included  in  each  bar"scalculation. 


Our  final  measure  of  how  teachers  spend  their  time  is  the  number  of  hours  per  week 
spent  on  administrative  or  disciplinary  tasks.  These  program-specific  averages  are 
displayed  in  Figure  15.  The  sample  averages  (not  shown)  are  1.5  for  administrative 
tasks  and  1.9  for  disciplinary  tasks  (for  a  total  of  3.4  so-called  nonproductive  hours 
per  week,  on  average).  In  a  number  of  programs,  as  shown,  teachers’  average  total 
nonproductive  hours  far  exceed  this,  including  Georgia-Fort  Stewart,  Illinois,  Hawaii- 
Kapolei,  and  Kentucky-Bluegrass,  all  of  which  have  an  average  of  5  nonproductive 
hours  per  week  or  more.  At  the  other  extreme  are  Washington,  Wisconsin,  Idaho, 
California-Sunburst,  New  Mexico,  and  North  Carolina,  all  of  whose  teachers  have  less 
than  2  nonproductive  hours  per  week.  This  variation  led  us  to  suspect  a  potential 
negative  impact  of  nonproductive  hours  on  cadets’  average  outcome  measures,  since 
any  additional  hours  that  teachers  must  devote  to  administrative  tasks  or  discipline 
are  hours  during  which  they  are  not  covering  pertinent  subject  matter.  As  has  been 
the  case  for  many  of  the  other  variables  in  this  section,  however,  we  find  no 
significant  relationship  between  these  nonproductive  hours  and  cadets’  outcomes  at 
the  program  level. 


31 


CNA 

Y^i*^  a  ^rw  t  jTinN^? 


Figure  15.  Average  hoursteachersspend  perweekon  administrative  and 
disciplinary  tasks,  byChalleNGe  program^ 
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Source:  CNA  tabulationsof  tea  c  her  survey  data. 

The  black  line  in  the  figure  correspondsto  the  secondary  vertical  axisand  represents  the 
number  of  responses  included  in  each  bar's  calculation. 


Pedagogical  methods 

We  now  turn  to  pedagogical  methods,  which  encompass  teachers’  preparation  of 
their  course  materials,  the  amount  of  homework  they  typically  assign,  and  whether 
they  tailor  their  course  materials  for  specific  students  as  needed.  Figure  16  displays, 
by  ChalleNGe  program,  the  distribution  of  teachers’  responses  to  who  prepared  their 
daily  course  materials.  The  options  provided  were  self -preparation,  joint  preparation 
with  other  ChalleNGe  staff,  purchased  materials,  a  mix  of  self-developed  and 
purchased  materials,  and  other. In  the  overall  sample,  the  majority  of  teachers  (53 
percent)  indicated  that  they  use  some  mix  of  purchased  and  self-prepared  materials. 


“Purchased  materials”  here  refers  to  teachers’  use  of  materials  purchased  by  their  ChalleNGe 
program,  not  materials  purchased  by  the  teachers  themselves. 
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The  remaining  47  percent  are  divided  as  follows:  20  percent  self -prepared,  11 
percent  jointly  prepared,  13  percent  purchased,  and  3  percent  other. 


Figure  16.  Distribution  of  teachers’  course  material  preparation  methods,  by 
ChalleNGe  program 


Purchased  ■  Mix  (purchased  and  self-prep) 

■  Other 


Source:  CNA  tabulationsof  teachersurvey  data. 


A  few  Interesting  outliers  observable  in  Figure  16  are  the  high  prevalence  of  teachers 
using  only  self-prepared  materials  in  Hawaii-Kapolei,  Idaho,  Oregon,  and  Puerto  Rico 
(all  60  percent  or  greater)  and  the  prevalence  of  teachers  using  purchased  materials 
daily  in  Arkansas,  Georgia-Fort  Stewart,  Louisiana-Gilhs  Long,  Mississippi,  and  West 
Virginia  (all  30  percent  or  greater).  As  with  the  other  pedagogical  variables,  we  find 
httle  correlation  between  who  prepared  teachers’  daily  course  materials  and  our 
outcome  measures.  The  one  exception  is  the  change  in  TABE  score:  programs  with  a 
higher  percentage  of  teachers  using  purchased  materials  are  also  programs  with  a 
higher  average  TABE  Improvement  among  their  cadets.  We  suspect  that  this  may  be 
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because  some  of  the  most  commonly  purchased  materials  are  standardized-test 
preparatory  materials.  This  is,  of  course,  only  a  supposition. 

We  find  significant  variation  across  ChalleNGe  programs  in  the  average  amount  of 
homework  teachers  assign  per  week.  We  asked  how  much  homework  teachers  assign 
in  a  typical  week,  and  possible  responses  included  none,  <1  hour,  1-2  hours,  3-4 
hours,  and  5  or  more  hours.  In  the  sample  as  a  whole,  the  three  most  common 
responses  were  no  homework,  less  than  1  hour,  and  1-2  hours,  each  accounting  for 
27  percent  of  responses.  Roughly  14  percent  of  teachers  indicated  that  they  assign 
3-4  hours  per  week,  and  the  remaining  4  percent  responded  that  they  assign  5  or 
more  hours  of  homework  in  a  typical  week.^^  We  found  the  number  of  teachers 
assigning  no  homework  as  well  as  the  number  assigning  5  or  more  hours  per  week  to 
be  surprising.  The  majority  of  teachers  (50  percent  or  more  of  survey  respondents) 
indicated  that  they  assign  no  homework  in  a  typical  week  at  the  following  programs: 
Hawaii-Kapolei,  Louisiana-Camp  Minden,  Michigan,  New  Mexico,  Oklahoma,  Texas, 
and  Wyoming.  At  the  other  extreme,  teachers  assigning  5  or  more  hours  of 
homework  were  concentrated  in  the  Georgia-Fort  Stewart,  Illinois,  Indiana, 
Mississippi,  and  Oregon  programs.  Even  with  the  program-level  variation  in 
homework  assignments  shown  in  Figure  17,  we  still  find  little  correlation  between 
these  homework  variables  and  cadet  outcomes,  with  one  exception.  Programs,  on 
average,  with  a  higher  concentration  of  teachers  assigning  less  than  1  hour  of 
homework  per  week  (compared  with  those  assigning  no  homework)  have  higher 
cadet  graduation  rates  and  higher  1 -month  placement  rates.  There  was  no 
statistically  significant  difference  in  cadet  outcomes  for  any  of  the  other  homework 
variables  (e.g.,  cadet  graduation  rates  at  programs  with  a  high  percentage  of  teachers 
assigning  1-2,  3-4,  or  5  or  more  hours  of  homework  are  indistinguishable  from  those 
with  a  high  percentage  of  teachers  assigning  no  homework).  These  findings  suggest 
that  cadets  respond  positively  to  a  small  and  manageable  number  of  tasks  to 
complete  outside  class,  which  perhaps  instills  a  sense  of  responsibility  and 
accountability. 


These  numbers  do  not  add  to  100  due  to  rounding. 
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Figure  17.  Average  homework hoursassigned  perweek,  byChalleNGe  program 


■  5  or  more  hours  ■3-4  hours  ■  1-2  hours  ■<lhour  ■  No  homework 


Source:  CNA  tabulationsof  tea  c  her  survey  data. 


The  final  question  we  asked  all  teachers  regarding  their  overall  teaching  approach 
was  how  frequently,  if  at  all,  they  change  their  instructional  methods  for  specific 
students.  On  one  hand,  we  might  expect  a  high  prevalence  of  “tailoring”  to  improve 
cadet  outcomes.  Varying  instructional  methods  allows  teachers  to  maximize  the 
“efficiency”  of  their  instruction  with  individual  cadets.  On  the  other  hand,  we  could 
imagine  a  scenario  in  which  excessive  “tailoring”  results  in  core  content  not  being 
conveyed.  To  answer  this  question,  we  coded  this  variable  to  take  a  value  of  0  for 
teachers  who  said  they  never  changed  their  instructional  methods  for  specific 
students,  1  for  those  who  did  so  rarely,  2  for  those  who  did  so  some  of  the  time,  and 
3  for  those  who  did  so  most  of  the  time.  The  average  value  of  this  variable,  by 
ChalleNGe  program  (along  with  the  national  average),  is  presented  in  Figure  18.  All 
the  program-specific  variation  occurs  between  values  of  1.5  and  2.5— a  rather  narrow 
range  (representing,  at  its  extremes,  teachers  who  fall  somewhere  between  “rarely” 
and  “some  of  the  time”  and  those  who  fall  between  “some  of  the  time”  and  “the 
majority  of  the  time”).  Similarly,  the  national  average  is  2.2.  Likely  as  a  result  of  this 
compressed  range  of  values,  we  find  no  statistically  significant  correlations  between 
the  frequency  of  “tailoring”  and  any  of  our  outcome  variables.  If  teachers  indicated 
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that  they  change  instructional  methods  for  specific  students,  we  asked  why  they  do 
so,  with  three  possible  responses:  (1)  students  are  at  different  places  academically; 
(2)  students  have  different  learning  styles;  and  (3)  other.  The  responses  were  46 
percent  for  option  1  and  52  percent  for  option  2;  only  2  percent  responded  with 
“other.”  Thus,  teachers’  responses  were  nearly  equally  divided  between  the  two 
options  we  provided. 


Figure  18.  Average  frequency  of  changing  instructional  methodsforspecific 
students,  by  ChalleNGe  program^ 
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Source:  CNA  tabulationsof  tea  c  her  survey  data. 

3- The  red  line  denotes  the  sample  average.  The  black  line  correspondsto  the  secondary 
vertical  axisand  represents  the  numberof  responses  included  in  each  bar"scalculation. 


Thus,  we  found  that  only  two  of  the  three  characteristics  of  “overall  teaching 
practices”  have  any  relationship  with  cadets’  outcomes  (whether  at  ChalleNGe  or 
longer  term).  Specifically, 

•  Programs  where  a  higher  percentage  of  teachers  used  purchased  course 
materials  as  their  primary  daily  material  showed  greater  average  TABE 
improvements.  We  suspect  this  is  because  the  programs’  purchased  materials 
most  likely  include  standardized-test  preparatory  materials. 

•  Programs  whose  teachers,  on  average,  assign  less  than  1  hour  of  homework 
per  week  have  higher  graduation  rates  and  1 -month  placement  rates.  This  is 
likely  due  to  the  fact  that  cadets’  days  are  highly  scheduled,  leaving  little  time 
to  dedicate  to  homework.  As  a  result,  the  noncognitive  skills  acquired  at 
ChalleNGe— such  as  independence  and  motivation— are  more  likely  to  prepare 
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them  for  college  and  employment  than  having  a  substantial  amount  of 
homework. 

These  findings  might  suggest  that  these  teaching  practices  should  considered  “best 
practices”  and  be  adopted  elsewhere.  As  previously  discussed,  however,  our 
analytical  framework— and  specifically  the  fact  that  we  are  analyzing  program-level 
averages  and  are  unable  to  match  teachers  to  cadets— is  not  rigorous  enough  to 
warrant  making  policy  decisions  based  on  these  findings.  An  individual-level  analysis 
would  be  necessary  to  draw  such  conclusions. 

Subject-specific  pedagogical  approaches 

In  addition  to  the  pedagogical  questions  already  summarized,  which  were  asked  of 
all  teachers,  our  survey  also  contained  a  subset  of  questions  asked  only  of  math 
teachers,  and  a  separate  subset  asked  only  of  English/language  arts  teachers.  We 
asked  additional  questions  of  these  two  groups  because  their  subjects  are  generally 
considered  the  two  core  curricular  areas.  This  is  especially  true  at  ChalleNGe,  since 
success  on  the  GED  and  other  standardized  tests  used  at  ChalleNGe  hinges  primarily 
on  skills  in  these  two  areas.  In  this  final  subsection,  we  characterize  the  pedagogies 
for  teachers  of  math  and  English/language  arts— with  greater  detail  on  the  specific 
content  covered  in  class— and  analyze  whether  these  pedagogies  are  related  to  any 
cadet  outcomes. 


Math 

A  third  (65  out  of  191)  of  our  survey  respondents  answered  the  questions  for  math 
teachers  only,  thus  identifying  themselves  as  math  teachers. We  asked  math 
teachers  how  often  they  address  specific  math  subjects— such  as  geometry,  algebra, 
and  learning  to  communicate  math  ideas  effectively— in  a  typical  math  class,  to 
determine  not  only  how  much  this  varies  across  programs  but  also  whether  the  math 
subjects  covered  affect  cadets’  outcomes.  Eor  analytical  purposes,  we  grouped  the 
specific  math  subjects  on  our  survey  into  four  aggregate  categories:  basic  math 
skills,  advanced  math  skills,  problem-solving  skills,  and  math  communication.^^  While 


This  number  is  higher  than  the  number  of  teachers  who  indicated,  earlier  in  the  survey,  that 
they  currently  teach  math  (59).  We  chose  to  retain  the  six  teachers  who  did  not  previously 
identify  themselves  as  such  in  the  sample  for  our  math-focused  analyses,  under  the 
assumption  that  although  they  might  not  currently  be  teaching  math,  they  at  some  time  taught 
have  math  in  the  ChalleNGe  program. 

Basic  math  skills  include  (among  others)  “whole  numbers  and  whole  number  operations”; 
advanced  math  skills  include  “data  analysis”;  problem  solving  includes  “developing 
reasoning/analytical  skills  to  solve  problems;  and  math  communication  includes  “learning  to 
communicate  math  ideas  effectively.” 
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there  is  some  variation  across  programs,  most  are  close  to  the  sample  averages  of  2.2 
hours  for  basic  math,  2.2  hours  for  advanced  math,  2.4  hours  for  problem  solving, 
and  2.2  hours  for  math  communication.  When  considering  the  impact  of  math 
subjects  and  the  extent  to  which  they  are  covered  in  a  typical  class  period,  we  do  find 
some  statistically  significant  relationships.  Namely,  basic  math  is  negatively 
correlated  with  programs’  graduation  rates,  1 -month  placement  rate,  6-month 
placement  rate,  and  12-month  placement  rate.  That  is,  programs  whose  math 
teachers  emphasize  basic  math  topics,  on  average,  have  lower  graduation  and  cadet 
placement  rates. In  addition,  we  find  a  positive  and  significant  relationship  between 
advanced  math  topics  and  cadets’  6-month  placement  rate.  It  is,  of  course,  possible 
that  the  amount  of  time  spent  on  different  math  topics  is  directly  related  to  student 
ability.  Teachers  whose  classes  are  composed  of  cadets  with  better  math  skills  may 
spend  more  time  on  advanced  topics  simply  because  there  is  less  need  to  focus  on 
basic  math  topics.  Thus,  this  finding  might  actually  be  related  only  to  cadets’  math 
abilities  (as  opposed  to  teachers’  choices  regarding  what  topics  to  cover),  and  it  may 
be  that  the  more  able  cadets  find  it  easier  to  maintain  their  focus  and  motivation, 
thus  increasing  their  probability  of  finishing  the  program.  Finally,  and  unexpectedly, 
we  find  a  negative  relationship  between  “problem-solving”  emphasis  and  the  12- 
month  placement  rate;  we  suspect  that  either  this  is  an  anomaly  (due  to  the  fact  that 
we’re  estimating  program-level  data)  or  it  reflects  a  trade-off  between  emphasizing 
“problem  solving”  or  “advanced  math,”  with  the  latter  being  more  important  to 
placement. 

We  also  asked  math  teachers  about  the  frequency  of  different  activity  types  in  their 
classrooms.  The  average  distribution  of  math-class  activities,  for  the  entire  sample,  is 
illustrated  in  Figure  19.  Not  surprisingly,  writing  reports/doing  math-related 
projects,  working  with  manipulative s,  and  working  with  measuring  instruments  are 
the  three  least  common  activities,  occurring  on  average  between  “never  or  hardly 
ever”  and  “once  or  twice  a  month.”  Among  the  most  common  activities  are  using  a 
calculator,  discussing  solutions  to  math  problems,  and  working  on  problems  that 
reflect  real-world  situations. 


As  we  presented  in  an  earlier  section  (Table  2),  only  9  percent  of  math  teachers  (nationally) 
majored  in  math.  As  a  result,  we  investigated  whether  this  negative  relationship  between  basic 
math  and  outcomes  is  being  driven  by  non-math-majors  teaching  math.  That  is,  are  these 
teachers  focusing  on  basic  math  because  they  lack  the  foundation  to  teach  more  advanced 
math  topics?  This  is  not  the  case:  there  is  no  statistically  significant  relationship  between  the 
prevalence  of  basic  math  topics  and  whether  math  teachers  majored  in  math. 
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Figure  19.  Frequency  of  math-class  a ctivities^ 


Work  on  math  "life  skills/'  e.g.  balancing  checkbooks 
Practice  by  teaching  or  explaining  to  each  other 
Take  practice  tests 
Use  a  computer 
Use  a  calculator 
Work  with  manipulatives 
Work  with  measuring  instruments 
Write  reports  or  do  math-related  projects 
Draw  a  picture  or  write  about  how  to  solve  math  probs. 

Discuss  solutions  to  math  problems 
Work  on  problems  in  small  groups  or  with  a  partner 
Work  on  problems  that  reflect  real-world  situations 
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Source:  CNA  tabulationsof  tea  c  her  survey  data. 

Activity  frequency  is  coded  as  follows:  0  is  “Never  or  hardly  ever,"  1  is  “Once  or  twice  a 
month,"  2  is“Once  ortwice  a  week,"  and  3  is  “Almost  every  day." 


We  also  estimated  whether  the  frequency  of  math-class  activities  is  correlated  with 
cadet  outcomes.  Table  5  summarizes  these  results.  Only  a  handful  of  results  are 
significant,  and  activities’  importance  (and  direction  of  effect)  varies  by  the  cadet 
outcome  being  considered.  Programs  whose  math  teachers  have  cadets  work  more 
frequently  with  measuring  instruments,  for  example,  tend  to  have  higher  graduation 
rates  but  lower  levels  of  TABE  improvement.  Another  counterintuitive  finding  is  the 
negative  relationship  between  working  on  real-world  problems  and  cadets’  average 
12-month  placement  rate.  As  previously  stated,  anomalies  such  as  this  may  be 
explained  by  the  fact  that  we  are  estimating  regressions  using  only  average-level 
data,  since  we  are  unable  to  match  specific  teachers  to  cadets.  The  two  most  striking 
findings  in  this  table  are  the  positive  and  significant  correlation  of  both  working  in 
small  groups  and  working  on  “math  life  skills”  with  multiple  positive  cadet 
outcomes.  This  may  suggest  that  these  two  activity  types  are  particularly  productive, 
not  only  in  improving  cadets’  math  skills,  but  also  in  preparing  them  in  other  ways 
for  life  after  ChalleNGe.  In  addition,  they  could  be  reflecting  underlying  reverse 
causality:  perhaps  the  cadets’  ability  levels  upon  entrance  (and  thus  likelihood  of 
achieving  TABE  improvement  or  graduating  from  ChalleNGe)  determine  the  activities 
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teachers  choose  to  assign  in  math  class,  as  opposed  to  the  activities  influencing  the 
cadets’  average  success  rates.  Without  individual-level  data,  we  carmot  dissect  this. 


Table  5.  Impactof  math-classactivitieson  cadetoutcome^ 


Program 

graduation 

rate 

TABE 

change 

Placement  rate 

Activity 

1 -month 

6-month  12-month 

Workon  problems  that  reflect 
real-world  situations 

Insig. 

Insig. 

Insig. 

Insig. 

<0 

Workon  problems  in  small  groups 
orwith  a  partner 

>0 

>0 

Insig. 

>0 

Insig. 

Discuss  solutions  to  math 
problems 

Insig. 

Insig. 

Insig. 

Insig. 

Insig. 

Draw  a  picture  orwrite  about 
how  to  solve  math  problems 

Insig. 

Insig. 

Insig. 

Insig. 

Insig. 

Write  reportsordo  math-related 
projects 

Insig. 

>0 

Insig. 

Insig. 

Insig. 

Work  with  measuring  instruments 

>0 

<0 

Insig. 

Insig. 

Insig. 

Work  with  manipulatives 

<0 

Insig. 

Insig. 

Insig. 

Insig. 

Use  a  calculator 

Insig. 

Insig. 

Insig. 

Insig. 

Insig. 

Use  a  computer 

Insig. 

Insig. 

Insig. 

Insig. 

Insig. 

Take  practice  tests 

Insig. 

Insig. 

Insig. 

Insig. 

Insig. 

Practice  by  teaching  or 
explaining  to  each  other 

Insig. 

Insig. 

Insig. 

Insig. 

Insig. 

Workon  “math  life  skills" (e.g., 
balancing  checkbooks) 

>0 

Insig. 

Insig. 

>0 

>0 

Source:  CNA  regnsssion  analysisof  teachersurvey  and  annual  report  data. 

"Insig."  indicates  that  the  relationship  between  that  math-class  activity  and  the  outcome 
variable  of  interest  is  statistic  ally  insignificant. 


The  final  three  questions  we  asked  math  teachers  were  what  types  of  assignments 
they  give  their  students,  how  adequate  they  feel  their  students’  math  skills  are  at  the 
beginning  of  the  program,  and  their  math  teaching  philosophies.  Figure  20  presents 
the  average  per-teacher  frequency  of  different  evaluation/assignment  types,  by 
program.  We  asked  how  frequently  teachers  use  each  of  the  following  in  their 
classes:  multiple-choice  tests,  problem  sets,  short  or  long  written  responses,  and 
Individual  or  group  projects  or  presentations.  The  possible  responses  were  “never  or 
hardly  ever,”  “a  few  times  during  the  course,”  “once  or  twice  a  month,”  and  “at  least 
once  a  week,”  and  these  were  assigned  values  of  0  to  3,  respectively.  The  maximum 
value  achievable  in  Figure  20  is  12— corresponding  to  a  value  of  3  for  each  of  the 
four  categories.  Teachers  at  most  programs  Indicated  using  a  combination  of  all  four 
evaluation  types.  There  are,  however,  a  number  of  outliers  (i.e.,  programs  whose 
math  teachers  Indicated  using  fewer  than  four  assigmnent  types):  District  of 
Columbia,  Idaho,  Michigan,  Montana,  New  Jersey,  North  Carolina,  Oklahoma,  South 
Carolina,  and  Texas. 
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Figure  20.  Average  per-teacherfnequency  of  evaluation  type,  byChalleNGe 
program^ 
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—  Projects/presentations  ■  Written  responses 


-  Problem  sets  ■  Multiple  choice  tests 

Source:  CNA  tabulationsof  teachersurvey  data. 

No  teachers  from  FL,  Hl-Kapolei,  IN,  orWI  responded  to  thisquestion. 


Because  of  this  variation— and  the  fact  that  not  all  math  teachers  use  all  four 
evaluation  types— we  investigated  whether  assignment  type  correlated  with  cadet 
outcomes.  This  analysis  revealed  only  one  significant  relationship:  programs’  average 
graduation  rates  are  negatively  correlated  with  the  prevalence  of  multiple-choice 
tests.  This  may  be  because  this  particular  evaluation  type  requires  only  that  students 
choose  the  correct  answer  from  a  list  of  possibilities  rather  than  engage  with  the 
material  the  way  the  other  three  evaluation  methods  require,  in  addition,  multiple- 
choice  tests  provide  students  the  correct  answer  (among  possible  wrong  answers),  so 
their  arriving  at  the  correct  answer  is  passive;  of  the  four  evaluation  types,  it  ehcits 
the  least  amount  of  critical  thinking  on  the  part  of  cadets.  We  recommend,  however, 
that  individual-level  analysis  be  conducted  to  confirm  this  relationship  before 
discouraging  the  use  of  multiple-choice  tests. 

Our  question  about  cadets’  math  competency  at  the  beginning  of  the  program  asked 
teachers  to  rate  separately  their  students’  skills  in  addition  and  subtraction. 
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multiplication,  and  division.  For  each  of  these  three  math  skills,  teachers  had  four 
options  from  which  to  choose  (which  we  then  assigned  values  of  0  to  3): 

•  No  students  are  very  competent 

•  Few  students  are  very  competent 

•  Some  students  are  very  competent 

•  Most  students  are  very  competent 

In  most  cases,  teachers  rated  their  students’  initial  math  skills  highest  in  addition 
and  subtraction,  followed  by  multiplication,  and  then  division.  As  we  would  expect, 
since  these  skills  build  in  complexity,  at  no  programs  did  teachers  rate  their 
students’  initial  skills  in  either  multiplication  or  division  higher  than  those  in 
addition  and  subtraction.  The  sample  averages  for  these  three  competencies  are  2.3 
for  addition/subtraction,  1.7  for  multiplication,  and  1.4  for  division  (where  an 
average  score  of  1.4,  for  example,  falls  between  “few  students  are  very  competent” 
and  “most  students  are  very  competent”— slightly  closer  to  the  former  than  the 
latter).  The  programs  that  stand  out  are  those  whose  students’  initial  math  skills  are 
either  substantially  higher  or  lower  than  the  sample  average.  For  addition  and 
subtraction,  these  include  the  various  programs  with  an  average  student  competency 
of  3  (above  the  sample  average)  as  well  as  Puerto  Rico,  South  Carolina,  New  Jersey, 
and  Georgia-Fort  Stewart  (below  the  sample  average). None  of  these  competency 
variables,  however,  are  significantly  correlated  with  cadets’  outcomes.  This  may 
suggest  that  teachers’  beliefs  are  not  entirely  accurate  or— as  we  have  found  in  other 
work  (see  [11])— that  cadets’  initial  skill  levels  are  not  important  predictors  of  their 
overall  program  success. 

Finally,  we  collected  information  on  math  teachers’  “teaching  philosophy.” 
Specifically,  teachers  were  asked  to  choose  which  of  the  following  statements  more 
closely  aligns  with  their  teaching  philosophies:^^ 

1.  Students  must  be  taught  math  skills  based  on  formulas  or  algorithms. 
Instruction  should  emphasize  procedural  mathematics  and  provide  step-by- 
step  examples  with  skill  exercises.  Focus  should  be  placed  on  the 
memorization  of  basic  math  facts  and  on  teaching  algorithms  that  efficiently 
lead  to  the  correct  answer.  (Formula-based  and  algorithmic  thinking) 


Those  programs  with  an  average  competency  of  3  for  addition  and  subtraction  are  California- 
Grizzly,  Idaho,  Kentucky-Bluegrass,  Louisiana-Camp  Minden,  Montana,  North  Carolina,  Texas, 
and  Washington. 

The  history  of  changes  in  math  education  in  the  U.S.  and  the  difference  between  these  two 
philosophies  is  discussed  in  more  detail  in  [12]  and  [13]. 
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2.  Instruction  should  be  focused  on  the  process  of  leading  to  the  correct  answer, 
rather  than  the  answer  itself.  Students  are  encouraged  to  represent,  describe, 
and  communicate  mathematical  ideas  by  a  variety  of  methods.  Conceptual 
understanding  is  the  primary  goal;  algorithmic  fluency  will  follow  secondarily. 
Students  should  learn  how  to  make  connections  between  concepts.  (Critical 
thinking) 

We  asked  teachers  to  choose  between  these  two  philosophies— while  acknowledging 
that  neither  statement  may  reflect  their  exact  philosophies— since  they  represent  the 
two  sides  of  an  ongoing  debate  about  how  math  should  be  taught  in  the  United 
States.  We  were,  of  course,  interested  not  only  in  how  the  ChalleNGe  teachers  align 
their  teaching  philosophies  with  these  two  statements,  but  also  whether  this  has  any 
relationship  to  programs’  average  cadet  outcomes.  Overall,  “critical  thinking”  was 
chosen  more  frequently  than  “formula-based  and  algorithmic  thinking”:  of  53  math 
teachers,  35  chose  the  former  and  18  chose  the  latter.  The  program-level  variation  in 
philosophies  is  displayed  in  Figure  21.  Programs  with  a  particular  emphasis  on  the 
“formula-based  and  algorithmic  thinking”  approach  Include  Arkansas,  Idaho, 
Kentucky-Bluegrass,  Louisiana-Camp  Minden,  Louisiana-Gilhs  Long,  Montana,  New 
Jersey,  Oregon,  and  Puerto  Rico.  These  programs’  average  math  philosophies  deviate 
significantly  from  the  sample  average,  leading  us  to  investigate  whether  the 
particular  math  philosophy  with  which  teachers  align  themselves  affects  cadet 
outcomes.  We  found,  however,  no  significant  relationship  between  average  math 
philosophy  and  cadet  outcomes  at  the  program  level  (including  program  graduation 
rate,  TABE  change,  and  postresidential  placement  rates). 

Of  all  the  subject-specific  information  gathered  from  math  teachers,  we  found 
significant  correlations  only  of  specific  math  subjects,  math-class  activities,  and 
assignment  types  with  cadet  outcomes,  hi  each  of  these  cases,  only  a  subset  of  the 
variables  was  significant  (e.g.,  “basic  math”  and  “advanced  math”  among  the  math 
subjects)  and  their  significance  varied  with  different  cadet  outcomes  (e.g.,  program 
graduation  rate  versus  1 -month  placement  rate).  We  did,  however,  find  preliminary 
evidence  that  the  math  subjects  emphasized,  the  type  of  activities  conducted  in  math 
class,  and  the  way  students  are  evaluated  in  math  are  correlated  with  longer  term 
measures  of  cadet  success.  This  suggests  that  more  robust,  individual-level  analysis 
should  be  conducted  in  this  area,  to  identify  how  to  most  effectively  structure 
ChalleNGe  math  curricula. 
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Figure  21.  Math  teachers'  philosophies,  by  ChalleNGe  program^ 


Source:  CNA  tabulationsof  tea  c  her  survey  data. 

No  teachers  from  FL,  Hl-Kapolei,  IN,  Wl,  orWV  responded  to  this  question. 

English/ language  arts 

About  a  third  (63  of  191)  of  our  survey  respondents  answered  the  questions  for 
English/language  arts  teachers  only,  thus  identifying  themselves  as  English/language 
arts  teachers. Similar  to  our  approach  for  math  teachers,  we  asked  how  much 
emphasis  these  teachers  place  on  different  topics— including  grammar,  literature, 
composition,  reading  comprehension,  study  skills,  and  spelling— in  order  to 
determine  how  their  subject-matter  emphases  vary  across  programs  and  whether 
these  are  correlated  with  cadets’  average  outcomes.  Eor  analytical  purposes,  we 
grouped  these  specific  subjects  into  two  aggregate  categories:  basic  English  subjects 
and  advanced  English  subjects.  The  basic  subjects  include  grammar,  reading 
comprehension,  study  skills,  and  spelling;  the  advanced  subjects  include  literature 
and  composition.  Possible  responses  for  each  of  the  six  subjects  listed  include  “not 


Earlier  in  the  survey,  126  teachers  indicated  that  they  currently  teach  English/language  arts. 
This  suggests  that  63  English/language  arts  teachers  chose  not  to  answer  the  subject-specific 
questions. 
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covered,”  “review  topic,”  “minor  topic,”  and  “major  topic.”  We  recoded  these 
variables  to  take  values  ranging  from  0  to  3,  with  0  representing  “not  covered”;  the 
program-level  averages  are  presented  in  Figure  22. 


Figure  22.  Emphasis  placed  on  basic  and  advanced  English/ language  artstopics, 
byChalleNGe  program 


3  T 


Source:  CNA  tabulationsof  tea  c  her  survey  data. 

3-  No  teachers  from  Hl-Kapolei,  NJ ,  NC,  SC,  WA,  orWV  responded  to  thisquestion.  ForHI- 
Kapolei,  SC,  WA  and  WV,  this  is  bee  a  use  no  teachers  identified  themselves  as 
English/ language  arts  teachers. 

The  yellow  line  re  presents  the  sample  average  for  basic  English/ language  arts  skills;  the 
red  line  represents  the  sample  average  foradvanced  skills. 


The  red  line  in  Figure  22  represents  the  sample  average  for  teachers’  emphasis  on 
advanced  subjects;  the  yellow  line  is  the  sample  average  for  emphasis  on  basic 
subjects.  At  most  programs,  teachers  emphasize  advanced  English/language  arts 
topics  over  basic  ones,  as  is  also  true  for  the  sample  as  a  whole. 

While  there  is  some  variation  across  programs,  most  are  close  to  the  sample  averages 
of  2.2  hours  for  basic  topics  and  2.4  hours  for  advanced  topics.  A  few  notable 
findings  include  the  particularly  high  emphasis  on  advanced  topics  at  the  Maryland, 
Michigan,  Oklahoma,  and  Wisconsin  programs;  the  low  emphasis  on  advanced  topics 
in  Hawaii-Hilo,  Oregon,  and  Puerto  Rico;  the  high  emphasis  on  basic  topics  in  Alaska, 
Georgia-Fort  Gordon,  Illinois,  Louisiana-Gillis  Long,  and  Wisconsin;  and  the  low 
emphasis  on  basic  topics  in  Florida,  Idaho,  Montana,  New  Mexico,  Puerto  Rico,  and 
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Virginia.  The  magnitude  of  English/language  arts  teachers’  average  emphasis  on 
basic  versus  advanced  topics,  however,  has  no  significant  correlation  with  any  of  the 
cadet  outcomes  (at  the  program  level). 

In  addition  to  grouping  English/language  arts  topics  into  basic  and  advanced,  we 
asked  teachers  about  their  strategies  for  teaching  English/language  arts  topics. 
Specifically,  we  asked  teachers  how  often  they  ask  their  students  to: 

•  Read  aloud 

•  Write  about  what  they  have  read 

•  Read  books  of  their  own  choice  in  class 

•  Do  a  group  activity  about  what  they  have  read 

•  Watch  movies  or  videos,  listen  to  CDs  or  tapes 

•  Make  predictions  about  what  they  are  reading  as  they  read 

•  Summarize  what  they  have  read 

Teachers’  possible  responses  are  “never  or  hardly  ever,”  “once  or  twice  a  month,” 
“once  or  twice  a  week,”  and  “almost  every  day,”  which  we  recoded  to  take  numerical 
values  ranging  from  0  to  3,  with  0  representing  “never  or  hardly  ever.”  The  average 
frequencies  for  each  activity  type  (for  the  sample  as  a  whole)  are  presented  in  Eigure 
23.  The  least  frequent  activities  are  watching  movies/videos  or  listening  to 
tapes/CDs  and  doing  a  group  activity  about  what  they  have  read.  On  average,  our 
survey  respondents  indicated  that  they  use  these  activity  types  in  their  classrooms 
only  “once  or  twice  a  month.”  The  most  common  activities  are  summarizing  what 
they  have  read,  reading  aloud,  and  reading  books  of  their  choice  during  class.  As  we 
have  done  with  other  pedagogical  methods,  we  estimated  the  relationship  between 
the  frequency  of  these  activity  types  and  cadet  outcomes.  Only  one  of  these  seven 
English/language  arts  classroom  activities  is  statistically  significantly  related  to  any 
cadet  outcomes:  reading  books  of  their  own  choice  in  class.  Specifically,  the  more 
often  teachers  asked  students  to  read  books  of  their  choosing,  the  higher  the  average 
6-month  and  12-month  postresidential  placement  rates.  This  suggests  that  this 
particular  English/language  arts  activity  has  long-term  impacts  on  cadets,  perhaps  by 
teaching  them  how  to  be  independent  and  possibly  improving  their  desire  to  learn. 
As  we  discussed  in  the  math  subsection,  it  is  possible  that  any  findings  of  positive 
association  between  English/language  arts  activities  and  cadet  outcomes  are  a  result 
of  reverse  causality.  If  English/language  arts  teachers  choose  their  classroom 
activities  based  on  cadets’  ability  levels  (i.e.,  what  they  think  the  cadets  can  manage), 
then  in  lieu  of  classroom  activities  influencing  the  likelihood  of  postresidential 
placement,  it  may  be  that  the  postresidential  placement  success  is  a  function  of 
cadets’  innate  abilities  (and  that  the  classroom  activities  were  chosen  based  on  these 
abilities).  These  possible  relationships  cannot  be  differentiated  without  individual- 
level  data. 
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Figure  23.  Frequency  of  English/ language  a  its  activities,  forthe  progrann  asa  whole 


Summarize  what  they  have  read 

Make  predictions  about  what  reading  while 
reading 

Watch  movies  or  videos,  listen  to  CDs  or 
tapes 

Do  group  activity  about  what  they  read 
Read  books  of  choice 
Write  about  reading 
Read  aloud 
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Source:  CNA  tabulationsof  tea  c  her  survey  data. 


We  also  asked  English/language  arts  teachers  how  many  books  they  require  cadets  to 
read  in  the  course  of  their  classes  (which  are  approximately  five  months  long)  and 
how  often  they  use  different  types  of  literature.  The  average  number  of  books  that 
English/language  arts  teachers  require  cadets  to  read,  by  program,  is  illustrated  in 
Eigure  24.  The  average  for  the  sample  as  a  whole,  shown  by  the  red  line,  is  2.3  books. 
The  program-specific  averages  vary  significantly,  from  a  maximum  of  6  books  in 
Idaho  to  no  books  at  Louisiana-Gillis  Long,  New  Mexico,  and  Puerto  Rico.  Regardless 
of  these  variations,  however,  the  average  number  of  books  that  cadets  are  required  to 
read  in  the  course  of  a  class  is  statistically  insignificantly  related  to  all  cadet 
outcome  variables.  That  is,  the  average  number  of  books  that  English/language  arts 
teachers  at  a  particular  ChalleNGe  program  require  their  cadets  to  read  is  not 
correlated  with  the  program’s  average  graduation  rate,  change  in  TABE  scores,  or 
postresidential  outcomes. 
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Figure  24.  Average  numberofbooksteachers require  cadetsto  read,  byChalleNGe 

program^b 


Source:  CNA  tabulationsof  teachersurvey  data. 

3  The  red  line  denotesthe  sample  average. 

The  squareson  the  x-axisdenote  those  programs  whose  English/ language  arts  teachers 
indicated  thatthey  do  not  require  cadetsto  read  any  books.  The  otherprograms  with  no 
data  eitherhad  no  English/ language  artsteachersamong  respondents ortheir 
English/ language  arts  respondents  did  notanswerthisquestion. 


We  asked  teachers  how  frequently  they  used  different  types  of  literature,  including, 
fiction,  poetry,  mythology,  biography,  drama,  argument/persuasion,  procedural,  and 
other  nonfiction.  Teachers’  possible  responses  included  “never,”  “rarely,”  “some  of 
the  time,”  and  “majority  of  the  time”  (which  we  recoded  to  take  values  of  0,  1,  2,  and 
3,  respectively).  Figure  25  shows  that  the  type  of  literature  most  commonly  used  in 
English/language  arts  classes  is  argument/persuasion,  followed  by  fiction,  other 
nonfiction,  and  biography.  Mythology  is  the  most  infrequently  used  type,  followed  by 
poetry  and  drama.  We  also  analyzed  whether  the  program-level  variation  in  the 
frequency  of  types  of  literature  used  was  correlated  with  any  cadet  outcomes.  It  was 
not;  the  substitution  of  one  hterature  type  for  another  had  no  bearing,  on  average,  on 
cadets’  average  outcomes  at  a  particular  program. 
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Figure  25.  Average  frequency  of  literature  typesused,  allChalleNGe  programs 


Source:  CNA  tabulationsof  tea  c  her  survey  data. 


We  also  asked  teachers  about  the  particular  processes  they  emphasize  when  reading 
informational  and  library  texts  in  class.  We  specifically  asked  about  the  frequency  of 
three  different  processes:  locate/recall  material,  integrate/interpret  material,  and 
critique/evaluate  material.  Possible  responses  included  “not  at  all,”  “to  a  small 
extent,”  “to  a  moderate  extent,”  and  “to  a  large  extent,”  to  which  we  applied 
respective  values  of  0,  1,  2,  and  3.  The  average  frequencies  of  these  reading 
processes  are  presented,  by  program,  in  Figure  26.  In  the  sample  as  a  whole, 
teachers’  average  emphasis  on  locate/recall  was  2.22,  indicating  that  the  average 
teacher  emphasized  these  topics  “to  a  moderate  extent.”  The  sample  averages  for 
integrate/interpret  and  critique/evaluate  were  2.41  and  2.22,  respectively.  We  were 
particularly  interested  in  the  number  of  programs  whose  teachers  indicated 
emphasizing  critique/evaluate,  since  this  process  might  be  considered  the  most 
advanced  level  of  engaging  with  the  material.  As  Figure  26  illustrates,  teachers’ 
indicated  that  they  emphasized  critique/evaluate  to  a  large  extent  at  a  number  of 
programs,  including  Arkansas,  District  of  Columbia,  Florida,  Idaho,  Maryland, 
Michigan,  New  Mexico,  Puerto  Rico,  Virginia,  and  Wisconsin.  Our  analysis  of  the 
correlation  between  these  three  different  categories  of  emphasis  and  cadet  outcomes 
revealed  the  following: 

•  The  emphasis  placed  on  critique/evaluate  is  positively  and  statistically 
significantly  associated  with  cadets’  6-month  placement  rates,  at  the  program 
level.  Specifically,  a  one-unit  increase  in  this  variable  (indicating  a  change,  for 
example,  from  “not  at  all”  to  “to  a  small  extent”  or  from  “to  a  moderate 
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extent”  to  “to  a  large  extent”)  is  associated  with  an  increase,  by  20  percentage 
points,  of  cadets’  average  6-month  placement  rate  at  that  program. 

•  The  average  emphasis  placed  on  critique/evaluate  is  also  positively  and 
significantly  correlated  with  cadets’  average  12-month  placement  rate.  In  this 
case,  a  one-unit  increase  in  this  variable  is  associated  with  an  increase  in  the 
average  12-month  placement  rate  of  32  percentage  points. 

•  The  emphasis  placed  on  integrate/interpret  is  negatively  and  statistically 
significantly  associated  with  cadets’  12-month  placement  rate.  A  one-unit 
Increase  in  the  average  emphasis  that  a  program’s  teachers  place  on 
rntegrating/interpreting  text  is  correlated  with  a  14-percentage-pornt 
decrease  hi  the  12-month  placement  rate.  This  counterintuitive  result  may 
indicate  a  substitution  effect— perhaps  an  increased  emphasis  on 
rntegrate/interpret  results  in  a  decreased  emphasis  on  critique/evaluate  and 
the  latter  is  more  important  for  placement  rates. 


Figure  26.  Reading  emphasis,  byChalleNGe  program^ 
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Source:  CNA  tabulationsof  tea  c  her  survey  data. 

^■The  programs  with  no  data  either  had  no  English/ language  artsteachensamong 
respondentsortheir  English/language  arts  respondentsdid  not  a nswerthis question. 


In  addition  to  their  reading-related  pedagogical  methods,  we  asked  English/language 
arts  teachers  two  questions  about  their  approaches  regarding  writing.  In  particular: 
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How  often  do  you  ask  your  students  to  write  for  each  of  the  following 
purposes?  The  purposes  included  to  explain,  to  persuade,  to  convey 
experience,  and  to  summarize/synthesize  material  they  have  read.  Possible 
responses  were  “never  or  hardly  ever,”  “a  few  times  during  the  course,”  “once 
or  twice  a  month,”  and  “at  least  once  a  week.”  These  responses  were  then 
assigned  values  of  0  to  3,  respectively. 

2.  When  you  are  teaching  writing,  what  percentage  of  your  instructional  time  is 
spent  on  each  of  the  following?  Categories  listed  included  development  of 
ideas,  organization  of  ideas,  effectiveness  of  expression,  and  mechanics  and 
convention.  Teachers  chose  from  percentage  intervals:  0  to  24  percent,  25  to 
49  percent,  50  to  74  percent,  75  to  89  percent,  and  90+  percent. 

Regarding  these  two  questions,  we  evaluated  both  teachers’  responses  and  whether 
they  are  related  to  any  of  the  cadet  outcome  variables.  In  terms  of  the  frequency  of 
different  writing  purposes,  the  average  responses  did  not  differ  much  from  across 
writing  purposes  in  the  sample  overall.  Specifically,  the  average  frequencies  were 
2.16  for  “to  explain,”  1.98  for  “to  persuade,”  2  for  “to  convey  experience,”  and  2.27 
for  “to  summarize/synthesize.”  Thus,  for  each  writing  purpose,  the  average  response 
per  teacher  was  close  to  2,  representing  “once  or  twice  a  month.”  The  variation 
across  programs  is  displayed  in  Figure  27.  At  the  two  extremes,  teachers  surveyed 
from  the  Arkansas,  District  of  Columbia,  and  Virginia  programs  responded  “at  least 
once  a  week”  for  every  writing  purpose,  whereas  teachers  at  the  Wyoming  program 
indicated  that  they  “never  or  hardly  ever”  have  students  write  “to  summarize,”  and 
those  from  Idaho  responded  “a  few  times  during  the  course”  for  each  of  the  four 
writing  categories.  In  terms  of  correlation  of  the  frequencies  of  different  writing 
purposes  with  cadets’  average  outcomes,  we  find  two  statistically  significant 
relationships.  First,  how  often  teachers  have  students  write  to  summarize  is 
positively  correlated  with  both  cadets’  average  6-month  and  12-month  placement 
rates.  Conversely,  how  often  teachers  have  students  write  to  explain  is  negatively 
associated  with  both  the  average  6-month  and  12-month  placement  rates.  This 
counterintuitive  finding  may  reflect  a  substitution  effect.  It  may  be  that  teachers  who 
have  students  write  to  explain  more  often  have  them  write  to  summarize  less  often, 
and  the  latter  is  positively  correlated  with  the  6-  and  12-month  placement  rates. 
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Figure  27.  Average  per-teacherfnequency  of  writing  purposes,  byChalleNGe 
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Source:  CNA  tabulationsof  tea  c  her  survey  data. 

^■The  programs  with  no  data  either  had  no  English/ language  artsteachensamong 
respondentsortheir  English/language  arts  respondentsdid  not  a nswerthis question. 

b-The  blackline  correspondsto  the  secondary  vertical  axisand  represents  the  numberof 
responses  included  in  each  bar^scalculation. 


Of  all  the  subject-specific  information  gathered  from  English/language  arts  teachers, 
we  found  significant  correlations  of  only  the  most-emphasized  reading  processes 
and  the  frequency  with  which  teachers  have  their  cadets  write  for  different  purposes. 
In  each  of  these  cases,  only  a  subset  of  the  variables  was  significant  (e.g., 
critique/evaluate  and  integrate/interpret  among  the  reading  processes),  and  their 
significance  varied  with  different  cadet  outcomes  (e.g.,  program  graduation  rate 
versus  1 -month  placement  rate).  We  do,  however,  find  preliminary  evidence  that  the 
reading  processes  emphasized  and  the  frequency  with  which  cadets  are  asked  to 
write  for  different  purposes  are  correlated  with  longer  term  measures  of  cadet 
success.  This  suggests  that  more  robust,  individual-level  analysis  should  be 
conducted  in  this  area,  to  identify  how  to  most  effectively  structure  ChalleNGe 
English/language  arts  curricula. 
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Recommended  Classroom  Strategies 


The  survey  gave  respondents  an  opportunity  to  share  information  about  the  teaching 
or  classroom  management  techniques  they  have  found  to  be  particularly  effective. 
Our  goal  was  to  compile  a  list  of  the  most  frequently  cited  “lessons  learned”  in  the 
hopes  they  could  be  adopted  by  other  ChalleNGe  teachers.  This  section  summarizes 
the  teachers’  responses. 

This  survey  question  was  open-ended,  allowing  teachers  to  write  as  much  as 
necessary  to  fully  answer  the  question.  Of  the  191  surveys  submitted,  71  responded 
to  this  question  and  included  96  separate  comments  or  ideas.  We  grouped  these 
comments  into  broad  categories.  Most  comments  (all  but  12)  contained  some  version 
of  the  following  recommendations: 

•  Be  flexible  with  your  teaching  methods:  Teachers  commented  that,  given 
the  often  broad  range  of  academic  abilities  of  their  students  on  arrival  at 
ChalleNGe,  successful  teachers  must  be  prepared  to  change  the  content 
and/or  delivery  of  their  lessons  to  meet  the  needs  of  the  students.  In  the 
words  of  one  teacher,  “teachers  need  a  wide  variety  of  tricks  in  their  bags  and 
[must]  be  flexible  about  where  and  when  to  use  them,  especially  when 
working  with  at-risk  youth.”  Teachers  also  noted  that  they  use  a  combination 
of  small-group,  individual,  and  whole-class  instruction  depending  on  the 
topic  and  the  level  of  their  students’  knowledge. 

•  Work  to  build  relationships  with  your  students:  Numerous  comments 
centered  on  teachers’  feelings  that,  especially  with  at-risk  youth,  developing 
student/teacher  relationships  based  on  trust  is  critical  to  students’  achieving 
academic  success.  The  students  need  to  know  that  their  teachers  care  about 
them  as  more  than  just  a  student.  “1  want  to  earn  the  students’  trust  so  they 
can  feel  relaxed  and  happy  in  class,”  said  one.  Teachers  also  felt  that  a  strong 
teacher/student  relationship  helped  minimize  behavioral  issues,  thus 
allowing  for  more  effective  classroom  management. 

•  Ensure  that  your  lessons  are  engaging  and  relevant:  Students  typically 
enter  the  ChalleNGe  program  with  negative  feelings  about  school  and 
learning.  To  combat  this  and  help  the  cadets  begin  to  believe  in  themselves 
and  their  ability  to  learn,  teachers  noted  that  using  engaging  lessons  that  are 
relevant  to  cadets’  lives  can  help  capture  their  attention  and  keep  them 
motivated.  This  includes  using  hands-on  projects,  project-based  learning,  and 
contextual  learning.  As  one  teacher  said,  “1  have  learned  that  the  most 


53 


CNA 

Y^i*^  a  ^rw  t  jTinN^? 


effective  way  of  teaching  the  cadets  is  by  adapting  the  material  to  tangible 
and  relevant  things  that  occur  or  have  been  experienced  in  their  daily  lives.” 

•  Set  clear  expectations:  To  help  students  succeed  academically,  teachers 
noted  the  importance  of  setting  clear  expectations  early  on  regarding  not  just 
academics,  but  also  behavior.  As  one  teacher  highlighted,  “A  number  of  these 
cadets  are  not  used  to  being  students.  That  is,  they  are  not  used  to  going  to 
school  every  day  and  going  to  every  [class]  period.”  The  cadets  need  to  be 
given  specific  guidelines  so  they  can  learn  to  be  good  students— a  skill  as 
important  as  the  academic  skills  being  taught.  Teachers  also  noted  that 
consistency  was  critical  in  the  enforcement  of  those  expectations.  If  cadets 
sense  that  they  can  get  away  with  breaking  the  “rules,”  they  will. 

•  Provide  sufficient  positive  reinforcement:  ChalleNGe  cadets  are  not 
accustomed  to  success  in  the  classroom  and  are  in  special  need  of  positive 
reinforcement  regarding  their  education.  They  need  to  believe  that  they  can 
learn  and  be  successful  academically.  Positive  reinforcement  should  include 
praise  for  doing  good  work,  incentives  for  going  beyond  the  required 
assignment,  and  reminders  that  they  are  capable  students.  Praising  cadets 
who  are  staying  on-task  can  also  help  to  maintain  classroom  discipline  since 
it  may  encourage  those  not  receiving  praise  (because  they  are  not  “on  task”) 
to  shift  their  focus  to  the  academic  tasks  at  hand.  These  youth  typically  have 
received  scant  praise  in  the  past,  especially  related  to  academic  progress; 
thus,  observing  their  peers  being  praised  can  serve  as  a  motivating  factor. 

As  noted  throughout  our  discussion  of  our  quantitative  findings,  in  many  cases  we 
did  not  find  a  statistically  significant  relationship  between  our  measures  of  teachers’ 
qualifications  or  pedagogies  and  cadets’  average  outcomes.  Although  this  may,  in 
part,  be  due  to  the  limitations  of  our  analysis— including  the  fact  that  we  cannot  link 
teachers  to  students  and  are  able  only  to  correlate  programs’  average  characteristics 
with  their  cadets’  average  outcomes,  the  lack  of  correlation  may  also  suggest  that 
unobservable  teacher  differences  affect  cadets’  outcomes.  The  recommended 
classroom  strategies  summarized  here  are  likely  candidates  for  unobservable  (i.e., 
not  surveyed)  pedagogies  that  could  significantly  affect  our  measures  of  cadets’ 
success,  both  in  the  ChalleNGe  classroom  and  beyond. 
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Conclusions  and  Recommendations 


The  primary  goal  of  this  study  was  to  enumerate  how  pedagogical  approaches  differ 
across  ChalleNGe  programs  and  to  investigate  whether  these  differences  are 
correlated  with  short-  and/or  long-term  cadet  outcomes,  including  program 
graduation  rate,  TABE  improvement,  and  postresidential  placement.  We  also 
attempted  to  identify  classroom  methods  that  are  especially  effective  with 
disadvantaged  youth  performing  below  grade-level.  To  do  so,  we  used  data  from  an 
online  survey  administered  to  all  ChalleNGe  teachers  in  addition  to  data  from  the 
ChalleNGe  programs’  annual  reports.  The  primary  contribution  of  this  report, 
however,  is  to  provide  the  ChalleNGe  program  with  information  on  differences  across 
the  various  sites  that  had  previously  been  undocumented. 

Due  to  the  lack  of  detailed  data,  our  analysis  has  some  limitations.  First,  we  cannot 
match  teachers  to  particular  cadets  because  all  ChalleNGe  cadets  are  under  the 
purview  of  all  teachers.  This  limits  our  analysis  because  individual-level  data  are 
necessary  to  confidently  establish  relationships  between  teacher  characteristics  and 
student  outcomes. 

Second,  information  on  our  outcome  measures  (program  graduation  rate,  TABE 
improvement,  and  postresidential  placement)  is  available  only  in  average  form  at  the 
program  level.  Thus,  we  cannot  correlate  specific  teachers’  pedagogical  approaches 
or  qualifications  with  their  students’  outcomes;  we  can  only  correlate  programs’ 
average  teacher  qualifications  and  pedagogical  approaches  with  the  programs’ 
average  cadet  outcomes. 

Third,  the  postresidential  placement  data  are  incomplete.  Some  cadets  are 
unreachable  after  completing  ChalleNGe  and  some  likely  refused  to  respond.  In 
addition,  the  postresidential  information  we  do  have  is  less  than  ideal.  Specifically, 
the  postresidential  placement  rates  we  use  are  calculated  by  dividing  the  number 
placed  by  1,  6,  or  12  months  by  the  total  number  of  graduates  in  that  class.  It  would 
be  more  accurate  to  divide  the  number  of  cadets  placed  by  the  number  of  cadets  who 
provided  postresidential  placement  information,  but  this  information  is  not 
available.  Further,  the  most  recent  postresidential  data  available  are  for  previous 
classes;  the  teachers  who  responded  to  our  survey  may  not  be  the  teachers 
responsible  instructing  those  cadets  whose  postresidential  outcomes  are  captured  in 
our  data.  Our  findings,  therefore,  are  most  accurate  for  those  programs  with  little 
teacher  turnover  since  the  postresidential  cadets  in  the  2014  reports  graduated.  With 
these  caveats  in  mind,  we  summarize  our  most  notable  findings  below. 
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Differences  across  programs 

Our  survey  provides,  for  the  first  time,  detailed  information  on  ChalleNGe  teachers’ 
qualifications  and  pedagogical  approaches,  both  as  a  whole  and  at  individual 
programs.  Our  sample  may  not  be  representative  of  the  entire  ChalleNGe  teacher 
population  or  the  teacher  population  at  any  individual  program  because  it  is 
dependent  on  which  teachers  chose  to  complete  the  survey.  That  said,  several 
teacher  characteristics  are  worth  noting. 

Most  ChalleNGe  teachers  are  either  English/language  arts  teachers  or  math  teachers. 
These  are  the  two  subject  areas  critical  for  obtaining  either  a  GED  or  a  high  school 
diploma  (which  is  why  our  survey  focuses  on  these  two  curricular  areas).  Eewer 
teachers  teach  science  or  social  studies,  making  it  possible  that  these  two  curricular 
areas  are  not  receiving  the  focus  they  should  in  the  ChalleNGe  classrooms.  While 
hiring  additional  staff  to  teach  these  topics  could  remedy  this,  funding  issues  at  the 
program  level  may  prevent  it. 

Our  analysis  of  teacher  qualifications  across  the  entire  sample  revealed  that 
ChalleNGe  teachers,  on  the  whole,  are  highly  qualified.  Eor  example,  45  percent  of 
teachers  have  more  than  a  bachelor’s  degree,  indicating  a  highly  qualified  teaching 
cohort.  We  did,  however,  find  significant  variation  in  the  distribution  of  teachers 
holding  more  than  a  bachelor’s  degree  across  programs:  10  programs  had  no 
teachers  who  completed  the  survey  holding  more  than  a  bachelor’s  degree.  The 
average  years  of  teaching  experience  for  the  entire  sample  is  14,  again  suggesting 
that  ChalleNGe  teachers  are,  on  the  whole,  very  experienced.  But,  as  was  the  case 
with  educational  qualifications,  years  of  experience  varied  widely  across  programs, 
with  some  programs  having  an  average  teacher  experience  of  more  than  20  years  and 
others  less  than  7.  The  percentage  of  teachers  who  hold  teaching  certificates,  another 
measure  of  overall  qualification,  is  also  high— both  across  the  sample  as  a  whole  and 
within  most  ChalleNGe  programs.  The  sample  average  is  78  percent,  and  the 
program-specific  average  is  100  percent  for  18  of  the  34  programs  whose  teachers 
participated  in  the  survey. 

Teacher  qualifications  and  cadet  outcomes 

We  sought  to  determine  if  the  characteristics  of  the  teacher  population  has  any 
bearing  on  cadet  outcome  measures.  With  respect  to  teacher  qualifications  (e.g., 
highest  degree  held,  years  of  teaching  experience,  teaching  certification,  teachers’ 
majors  and  coursework),  we  found  significant  variation  in  teacher  qualifications 
across  programs.  Only  three  characteristics  (in  the  aggregate),  however,  had 
significant  relationships  with  cadets’  average  outcomes.  The  percentage  of  a 
program’s  teachers  who  majored  in  what  they  teach  and  the  teachers’  average  overall 
teaching  experience  were  both  significantly  and  negatively  related  to  TABE 
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improvement.  This  could  indicate  that  older  teachers  and  those  with  degrees  in  the 
subjects  they  teach  are  either  less  focused  on  TABE  improvement  or  are  less  aware  of 
how  most  effectively  to  improve  cadets’  TABE  performance.  The  third  and  most 
important  teacher  qualification  metric  appears  to  be  the  percentage  of  teachers  at  a 
program  who  have  attained  more  than  a  bachelor’s  degree.  This  program 
characteristic  was  positively  and  significantly  associated  with  all  cadet  outcomes, 
except  the  average  graduation  rate.  Ten  of  the  34  programs  surveyed  have  no 
teachers  with  education  beyond  a  bachelor’s  degree  who  responded  to  our  survey. 
This  suggests  that  some  programs  (and  thereby  cadets)  might  gain  from  an  increased 
percentage  of  teachers  with  postgraduate  degrees.  We  recommend,  however,  that  any 
policy  change  based  on  this  (or  other)  findings  be  made  with  care  since  they  cannot 
be  confirmed  by  individual-level  analysis  (due  to  the  ChalleNGe  construct  that  all 
cadets  are  under  the  purview  of  all  teachers). 

Pedagogy  and  cadet  outcomes 

Our  analysis  focused  on  various  pedagogical  approaches,  including  how  cadets  are 
assigned  to  classes  and  to  groups  within  classes,  average  class  size,  how  teachers  use 
their  instructional  time,  and  their  overall  pedagogical  strategies.  Once  again,  only  a 
few  measures  had  significant  relationships  with  cadet  outcomes. 

First,  we  found  that  programs  where  male  and  female  cadets  are  consistently 
grouped  either  separately  or  together  have  higher  12-month  placement  rates  than 
programs  where  gender  composition  varies.  This  suggests  that  consistency  in  gender 
grouping  is  important  for  these  cadets. 

Second,  we  found  that  programs’  average  1-month  placement  rates  are  statistically 
significantly  negatively  correlated  with  the  number  of  hours  spent  instructing  the 
class  as  a  whole.  This  suggests  that  students  benefit,  in  the  long  run,  from  more  one- 
on-one  instruction  and  less  large-group  instruction. 

Lastly,  we  found  that  programs  with  a  higher  percentage  of  teachers  using  purchased 
materials  tend  to  have  a  higher  average  TABE  improvement  among  their  cadets  and 
that  programs  with  a  higher  concentration  of  teachers  assigning  less  than  1  hour  of 
homework  per  week  (versus  those  assigning  no  homework)  have  higher  cadet 
graduation  rates  and  higher  1-month  placement  rates.  We  suspect  that  this 
relationship  to  TABE  improvement  may  be  because  some  of  the  most  commonly 
purchased  materials  are  standardized-test  preparatory  materials.  The  homework- 
related  findings  suggest  that  cadets  respond  positively  to  a  small  and  manageable 
number  of  tasks  to  complete  outside  the  classroom,  perhaps  Instilling  a  sense  of 
responsibility  and  accountability.  We  recommend  that  teachers  or  programs  not 
currently  assigning  homework  consider  doing  so. 
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Our  survey  also  included  subject-specific  questions  for  math  and  Enghsh/language 
arts  teachers  only.  We  focused  on  these  two  subject  areas  because  they  are  the  most 
common  focus  of  competency  exams,  such  as  the  GED. 

Math 

Our  analysis  of  math  teachers’  classroom  strategies  revealed  several  significant 
relationships  between  pedagogical  practices  and  average  cadet  outcomes.  When 
considering  the  impact  of  specific  math  subjects  and  the  extent  to  which  they  are 
covered  in  a  typical  class  period,  for  example,  we  found  that  focusing  instruction  on 
basic  math  is  negatively  correlated  with  programs’  graduation  rates,  1 -month 
placement  rate,  6-month  placement  rate,  and  12 -month  placement  rate.  That  is, 
programs  whose  math  teachers  emphasize  basic  math  topics,  on  average,  have  lower 
graduation  and  cadet  placement  rates.  We  also  found  a  positive  and  significant 
relationship  between  a  focus  on  advanced  math  topics  and  cadets’  6-month 
placement  rate.  This  suggests  that  cadets  might  be  better  served  by  including  more 
advanced  topics  in  their  math  curricula,  perhaps  because  they  are  more  engaging 
and/or  increase  cadets’  confidence  in  their  ability  to  understand  math  concepts. 

We  also  looked  into  whether  the  frequency  of  math-class  activities  is  correlated  with 
cadet  outcomes.  Only  a  handful  of  results  are  significant,  and  the  importance  the 
activities  varies  by  the  cadet  outcome  being  considered.  The  two  most  striking 
findings  are  the  positive  and  significant  correlation  of  working  in  small  groups  and 
teaching  math  life  skills  with  multiple  positive  cadet  outcomes.  It  is  likely  that  these 
teaching  strategies  are  preparing  cadets  for  life  after  ChalleNGe  (in  terms  of  their 
postresidential  placement),  indicating  that  teachers  should  consider  maximizing 
their  use  of  these  teaching  techniques. 

Finally,  we  found  that  average  graduation  rates  are  negatively  correlated  with  the 
prevalence  of  multiple-choice  tests,  suggesting  that  this  evaluation  method  should  be 
deemphasized. 

English/ language  arts 

We  found  several  statistically  significant  relationships  between  English/language  arts 
instructional  strategies  and  cadet  outcomes. 

First,  with  respect  to  reading  instruction,  emphasis  placed  on  critiquing  and/or 
evaluating  text  is  positively  and  statistically  significantly  associated  with  cadets’  6- 
month  placement  rates,  at  the  program  level.  Specifically,  a  one-unit  increase  in  this 
variable  is  associated  with  a  20-percentage-point  increase  in  cadets’  average  6-month 
placement  rate  at  that  program.  The  average  emphasis  placed  on  critiquing  and/or 
evaluating  text  is  also  positively  and  significantly  correlated  with  cadets’  average  12- 
month  placement  rate.  In  this  case,  a  one-unit  increase  in  this  variable  is  associated 
with  an  increase  in  the  average  12-month  placement  rate  of  32  percentage  points.  We 
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recommend  that  English/language  arts  teachers  consider  incorporating  critiquing 
and/or  evaluating  text  into  their  lesson  plans,  as  appropriate. 

We  also  found  that  the  frequency  of  assigning  writing  to  summarize  is  positively 
correlated  with  cadets’  average  6-month  and  12-month  placement  rates.  Conversely, 
how  often  teachers  have  students  write  to  explain  is  negatively  associated  with  the 
average  6-month  and  12 -month  placement  rates.  These  findings  suggest  that  using 
pedagogical  methods  that  prompt  students  to  interact  with  texts  they  are  reading 
and  to  engage  in  more  thoughtful  writing  have  important  implications  for  cadets’ 
success,  both  within  and  outside  ChalleNGe. 

A  number  of  caveats  accompany  this  analysis.  The  analysis  is  somewhat  limited  by 
the  ChalleNGe  construct— which  is  that  all  cadets  are  in  the  purview  of  all  teachers. 
Thus  we  cannot  match  students  to  individual  teachers  but  can  only  measure  cadet 
outcomes  at  the  average  program  level.  The  real  value  of  this  study  is  its 
documentation  of  the  differences  across  ChalleNGe  programs  and  its  identification 
of  how  these  differences  might  impact  cadets’  success  (both  within  and  after 
ChalleNGe).  Although  we  did  find  some  statistically  significant  relationships  between 
pedagogical  approaches  and  cadets’  average  outcomes,  we  reiterate  that  statistical 
significance  was  somewhat  sporadic.  These  facts  suggest  that  any  changes  in 
response  to  our  findings  should  be  made  only  after  further  research.  From  a 
programmatic  standpoint,  it  may  be  more  efficient  to  continue  to  allow  program 
directors  to  decide  how  to  structure  their  programs  rather  than  demand  that  all 
programs  adopt  certain  teaching  strategies  or  classroom  styles. 
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Appendix:  Teacher  Survey 


This  appendix  provides  the  complete  teacher  survey.  Most  teachers  took  the  survey 
online,  using  the  Checkbox  system,  rather  than  responding  on  paper. 
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How  Differences  in  Pedagogical  Methods  Impact  ChalleNGe  Program  Outcomes 

We  invite  you  to  participate  in  a  research  study  conducted  by  CNA  on  behalf  of  the  Office  of  the  Assistant 
Secretary  of  Defense  for  Reserve  Affairs.  Your  participation  in  this  study  is  completely  voluntary.  The 
following  is  a  brief  description  of  the  study  and  directions  on  how  to  complete  a  short,  on-line  survey.  If 
you  have  any  questions  about  the  study  or  the  survey,  please  contact  Dr.  Lauren  Malone,  Principal 
Investigator,  CNA,  at  malonel@cna.orq  or  (703)  824-2741. 

PURPOSE  OF  THE  STUDY 

CNA  is  a  not-for-profit,  federally-funded  research  and  development  corporation  that  does  research  on  a 
wide  range  of  topics  for  the  Department  of  Defense  and  other  Federal  agencies.  One  of  the  areas  in  which 
we  conduct  research  is  education.  CNA  has  been  asked  by  the  Office  of  the  Assistant  Secretary  of  Defense 
for  Reserve  Affairs  to  undertake  a  study  on  the  ChalleNGe  program.  Specifically,  they  have  asked  us  to 
look  at  the  various  ways  ChalleNGe  teachers  approach  instruction  with  the  objective  of  identifying  those 
pedagogical  methods  and  techniques  that  are  most  effective.  A  final  goal  of  the  project  is  to  share  the 
results  across  the  ChalleNGe  program. 

THE  SURVEY 

In  order  to  conduct  this  study,  we  are  collecting  data  from  all  teachers  at  all  of  the  ChalleNGe  sites  using 
this  on-line  survey.  First,  please  read  the  information  below  before  deciding  whether  or  not  to  participate. 
If  you  have  any  questions  about  anything  you  do  not  understand  in  this  survey  instrument  or  about  this 
research  study,  please  contact  Dr.  Malone,  whose  contact  information  is  listed  above,  before  proceeding. 
After  each  section,  please  check  the  box  to  indicate  your  agreement  with  the  statement. 

Process  and  Duration 

Your  participation  in  this  study  will  involve  your  completing  a  short  on-line  survey.  Most  items  are  multiple  choice,  but  a  few 
ask  you  to  fill  in  information.  It  should  take  approximately  20  minutes.  Your  participation  is  completely  voluntary.  You  have 
the  right  to  refuse  to  participate  and  doing  so  will  have  no  impact  on  your  employment  with  the  ChalleNGe  program.  If  you 
choose  to  partiapate,  you  may  also  skip  any  questions  you  don't  feel  comfortable  answering  or  stop  completing  the  survey 
altogether  at  any  point. 

understand  my  participation  is  completely  voluntary  and  that  if  I  choose  to  refuse  to 
participate,  it  will  have  no  impact  on  my  employment  with  the  ChalleNGe  program.  If  I  choose 
to  proceed,  I  understaiKl  that  I  may  skip  any  questions  I  do  not  feel  comfortable  answering  or 
stop  completing  the  survey, 
r  Yes 

r  No 
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Risks  and  Benefits 

There  are  no  risks  to  you  for  participating  in  this  study  and  we  do  not  believe  that  you  will  experience  any 
discomfort  in  answering  the  questions.  In  addition,  you  will  receive  no  direct  benefit  from  participation  in 
this  study.  However,  your  participation  may  help  the  National  Guard  Youth  ChalleNGe  program  to 
determine  what  changes  should  be  made  to  the  program  to  increase  positive  outcomes  for  cadets.  This 
could  play  a  role  in  increasing  General  Educational  Development  (GED)  passage  rates  and/or  success  for 
cadets  in  the  job  market. 

*  I  understand  there  are  no  risks  to  me  for  participating  in  this  study  and  that  I  will  receive  no 
direct  benefits  from  my  participation, 
r  Yes 
r  No 
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ChalleNGe  Survey 

Data  Secunty  and  Confidentiality 

Your  survey  responses  will  be  anonymous.  When  the  i^esuits  of  the  r^esear^  are  published  or  discussed  in 
conferences,  no  information  will  be  included  that  would  reveal  your  identity.  In  addition,  data  summaries 
will  aggregate  individual  responses  in  groups  sufficiently  large  enough  to  protect  the  anonymity  of 
individual  survey  respondents.  This  survey  is  being  administered  using  an  on-line  format.  None  of  the 
questions  will  ask  for  any  pei'sonal  data  (you  will  not  be  asked  to  give  your  name,  age,  etc.).  The  email 
address  used  to  invite  you  to  participate  in  the  survey  is  not  captured  when  you  complete  the  survey; 
there  is  no  link  between  your  email  address  and  your  completed  survey.  Your  completed  survey  will  be 
assigned  a  random  respondent  code  that  will  be  used  in  place  of  personal  identifier's  when  your  responses 
are  stored  by  CNA  and  in  ail  analyses  CNA  performs.  The  data  will  reside  on  secure  servers  that  are 
password  protected.  Access  is  granted  on  a  need  to  know  basis  only. 

understand  that  my  responses  will  be  anonynKHJS  and  that  data  collected  through  the  on-line 
survey  will  be  kept  secure, 
r  Yes 
r  No 
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Agency  DHciosure  Ststentent 

The  pubNc  reporting  burden  for  this  collection  of  Informaition  is  esbmated  to  average  20  minutes  per  response,  including  the 
time  for  reviewing  instructions,  searching  existirig  data  sources,  gathering  arnl  maintaining  the  data  needed,  and  completing 
and  reviewing  the  codection  of  mrormadon.  Send  commeres  regardir>g  this  burden  estimate  or  any  other  aspect  of  this 
collection  of  information,  indudirtg  suggestions  for  redudrig  the  burden,  to  the  Department  of  Defense,  Washington 
Headquarters  Services,  ExecuOve  Services  Directorate,  Information  Har^agement  Division,  4000  Mark  Center  Drive,  East 
Tower,  Suite  02G09.  Alexandria,  VA  22350-3100  (0MB  No.:  0704-0506).  Respondents  should  be  aware  that  notwithstanding 
any  other  provision  of  law,  r>o  person  shall  be  subject  to  any  penalty  for  faair>g  to  comply  with  a  collection  of  information  V  R 
does  not  display  a  currently  valid  OMB  control  number. 


*  Cons^t 

I  freely  and  voluntarily  consent  to  |>roceeding  with  the  on-line  survey, 
r  Yes 

r  No 
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Survey  Directions 

For  each  question,  choose  the  answer  that  best  fits.  You  may  skip  any  questions  you  do  not  want  to 
answer.  At  the  end  of  the  survey,  you  will  be  asked  to  dick  the  button  that  reads  '^Submit.*  This  will  take 
you  to  a  paqe  that  confinns  your  completion  of  the  suivey.  If  you  have  any  questions  or  experience  any 
difficulties  in  compiebng  the  survey,  please  contact  Jen  Atkin,  Research  Scientist,  CNA,  at  atkir^iaicna.Qi-g 
or  (703)  407-1048. 


Thank  you  for  helping  us  with  this  important  project. 
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At  which  ChalleNGe  program  do  you  teach? 

(Select: 

Are  you  a  lead  instructor? 
r  Yes 
r  No 

Do  your  male  and  female  cadets  take  classes  together  or  separately? 
r  Separately^  males  and  females  are  non  in  my  dassroom  at  the  same  time 
r  Together,  males  and  females  are  In  my  classroom  at  the  sanf>e  time 
r  Sonte  of  my  classes  indude  n^ales  and  females  together,  others  do  not 
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Other  than  the  subject(s)  yon  currently  teach  at  ChalleNGe,  which  other  subjects  have  you 
taught  in  the  past? 


How  many  years  of  teaching  experience  do  you  have  in  each  of  the  following  categories 
(including  this  year)? 

Enter  a  zero  where  applicaNe. _ 

No.  oryMTB 

Total  years  of  experience:  [ 

Years  teaching  at  ChalleNGe:  [ 

Years  teaching  grades  6-12:  | 

Years  teaching  in  alternative  programs  (su<t)  as  Job  Corps,  aRemative  schools,  and  ChalleNGe):  | 

Years  teaching  any  other  GED  prep  program:  [ 
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What  credentials  or  degrees  do  you  hold? 

Please  mark  all  that  apply, 
n  No  degree  beyond  higb  school 
n  Associates  degree/vocationaf  ceruncation 
r  BA/BS 

r  ma/ms 

n  Education  spedallst/pforessional  d^jloma  requiring  at  least  one  year's  woric  beyond  a  Masters'  degree 
r  EdD/PhD 

n  other  professional  degree: 

Do  you  hold  a  teaching  certificate? 

Please  select  all  that  apply. 

n  Yes.  a  standard  state  certillcate  or  advanced  professional  oertmcate 
F  Yes.  national  board  oertiflcaUon 
P  Yes.  a  certificate  requiring  addKIonai  coursevvork 
n  Yes.  an  emergency  certHkate 

r  No 

If  you  have  a  college  degree  (BA/BS  or  higher),  what  was  your  major  field  of  study  in  your 

highest  degree  program? 


If  applicable,  what  was  your  nvnor  field  of  study  in  college? 
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Have  you  taken  coursework  in  the  teaching  of  any  of  the  following  areas? 

Please  select  all  that  apply. 

F  Reading 

P  Any  other  language -arts-related  subject 
F  Benr>entary  or  seoorKSary  education 
F  Special  educaboo  (includir>g  students  with  disabuties) 

F  English  language  leamerVEngUsb  as  a  Second  Language  (ESL] 

F  AduR  education 
F  Mathematics 
F  Science 
r  Social  Studies 
F  None  of  the  above 

What  is  the  biggest  challenge  you  face  in  the  ChalleNGe  classroom? 

C  Weak  prior  mstniCbon/low  basic  level  of  achievement 
r  Cadets  dont  spend  enough  bme  at  ChalleNGe 
C  Cadets  dent  spend  enough  in  the  classroom 
r  Diverse  learners  in  one  classroom 
r  Disciplinary  issues  in  the  dassroom 
r  Other 

If  you  answered  Other  above,  please  explain  briefly  below: 
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Approximately  whet  percentepe  of  your  ChelleNGe  xtudents  pees  the  GED7 
(Leave  blank  if  you  don't  know  the  approximate  Agure.) 

I — 

Who  prepared  the  course  ruaterials  you  use  on  a  daily  basis? 

r  idkJ 

r  I  dkl,  wth  ocher  ChalleMGe  staff  members 
r  1  use  purchased  materials 

r  I  use  a  mix  of  seir-<Sevek>ped  and  purchased  materials 
r  other 

If  you  answered  Other  abovCr  please  explain; 


If  you  use  a  mix  of  seK-developed  and  purchased  materials,  please  indicate  the  percentage  you 
developed  or  helped  develop: 
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How  many  students  are  in  your  classroom  during  a  typical  class  period? 


If  your  class  size  varies  substantially,  please  tell  us  the  smallest  and  largest  classes  you  teach, 
and  the  reason(s)  for  the  variation: 
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PImm  answer  the  following  guasttons  based  on  your  axpariances  with  a  typical  class.  Por  axampla,  plaasa 
answer  time-based  questions  by  indicating  how  many  hours  per  week  you  spend  on  a  spedfVe  topic  or  sub|ect  in 
a  typical  dass.  If  dan  variation  makes  it  difficult  to  diooee  a  typical  dass,  please  answer  the  questions  based 
on  what  you  do  in  the  first  dass  you  teach  each  day. 


How  much  homework  do  you  assign  in  a  typical  week? 

r  None 

r  Less  than  1  hour  per  week 
r  1-2  hours  per  week 
C  3-4  hours  per  week 
r  hours  per  week 

How  are  students  in  your  desses  assigned  to  a  specific  dasa? 

Please  sdact  all  that  apply. 

^  AbdSy  or  previous  adiievement 

P  Age 

r~  Gervtor 

P  Ocher 

If  you  selected  Other  gbove,  pie—  apedfy; 


Do  you  arrange  or  create  groups  within  the  dass  based  on  ability  or  previous  achievement?  If 
so,  how? 
r  No 

r  Yes,  1  create  groups  based  on: 
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Instructional  time  is  limited;  deciding  how  to  spend  class  tinw  is  therefore  an  important 
determinant  of  student  outcon^ies.  How  nnich  class  tin^  do  you  spend  EACH  WEEK  on  the 

following  tasks? _ 

Houfi  p«r  wMk 


None 

<  1  hour 

1  hour 

2  hours 

3  hours 

4  hours 

hours 

Instructing  the 

dass  as  a  whole 

r 

r 

r 

r 

r 

r 

r 

Instructing  smal 
groups  or  students 

r 

r 

r 

r 

r 

r 

r 

Instructing  single 

students 

r 

r 

r 

r 

r 

r 

Discussing  issues 
as  a  dass 

c 

r 

r 

r 

r 

r 

r 

Assisting  students 
as  they  work  m 

groups 

r 

r 

r 

r 

r 

r 

r 

Assisting  students 
as  they  work 
independently 

r 

r 

r 

r 

r 

r 

r 

Assisting  students 
as  they  work  on 
homeworlc 

r 

r 

r 

r 

r 

r 

r 

Maintaing 

order/disdplining 

students 

r 

r 

r 

r 

r 

r 

r 

Administering 
tests  or  Quizzes 

r 

r 

r 

r 

r 

r 

r 

Administrative 
tasks  (I.e.,  taking 
attendance, 
making 

announcements, 

etc) 

r 

r 

r 

r 

r 

r 

r 

How  nxich  time  each  week  do  you  spend  tutoring  or  working  with  students  outside  of  class? 
Total  Ome  to  the  nearest  hour. 
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How  often  do  you  change  your  instructional  n>ethods  for  specific  students? 
r  The  majority  of  the  time 
r  Some  of  the  time 
r  Rarely 
r  Never 

If  you  change  your  instructional  methods  with  specific  students,  why  do  you  do  so? 
Please  select  all  that  apply. 

P  Students  are  at  different  places  academicaify 

P  Students  have  different  learning  styles 
r  Other 

If  you  selected  Other  above,  please  briefly  explain: 


Do  you  use  any  formal  n>ethod(s)  to  ascertain  students'  learning  styles? 

(  No 

r  Yes,  I  use: 

Is  there  anything  else  you  would  like  to  tell  us  about  your  teaching  or  classroom  management 
style  that  you  believe  to  be  particularly  effective  with  your  students? 
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For  Math  and  Language  Arts  teachers,  the  remaining  questions  on  the  survey  are  specific  to  the 
subject  matter  you  teach.  If  you  are  willing  to  proceed  and  answer  subject-specific  questions, 
please  indicate  the  subject  you  teach  below  and  additional  questions  will  appear. 

Which  subject(s)  do  you  teach? 

~  EngNsfx/Language  Arts 

r  Math 
r  Science 
r  Social  Studies 
r  rd  prefer  not  to  say 
r  Other 

If  you  answered  Other  above,  please  specify; 
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Mathematics  specific  questions 

Please  answer  the  following  questions  based  on  your  experiences  with  a  typical  Math  dass.  such  as  your 
9am  dass.  For  example,  please  answer  bme-based  questions  by  indicating  how  many  hours  per  week  you 
spend  on  a  specific  topic  or  subject  in  a  typical  dass.  If  dass  variation  makes  it  difficult  to  choose  a  typical 
class,  please  answer  the  questions  based  on  what  you  do  in  the  first  dass  you  teach  each  day. 


In  your  classes  how  often  do  you  address  the  following? 


Not  covered 

Review  topic 

Minor  topic 

Major  topic 

Wtiote  ntimbeis  and 

wibole  numbef 

operations 

r 

C 

r 

C 

ProMem  solving  a 
measurement,  using 
tables/ graphs 

r 

C 

r 

C 

Fractions/ percentage 
char>ges/proportk>ns 

r 

r 

r 

r 

Geometry 

r 

c 

r 

c 

Data  analysis 

r 

c 

c 

Statistics  and 

probability 

r 

r 

c 

r 

Algebra  (formulas 
and  equations) 

r 

r 

r 

r 

Developing 

reasoning/analytic 

skiMs  to  solve 
problems 

r 

r 

r 

r 

Learning  to 
communicate  math 
ideas  effectively 

r 

c 

r 

r 
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How  often  do  students  in  your  class  cio  each  of  the  following? 


Never  or 

Once  or  twice  a 

Once  or  twice  a 

Almost  every 

hardly  ever 

month 

week 

day 

Work  on  proMems  that 
reflect  reahworld 

sftuations 

r 

r 

r' 

r 

Work  on  problems  m 
smal  proops  or  with  a 
partner 

r 

r 

r' 

r 

Disajss  solutions  to 

math  problems 

r 

r 

r 

r 

Draw  a  picture  or  write 
about  how  to  solve 

math  problems 

r 

r 

r 

r 

Write  reports  or  do 
math-related  projects 

r 

r 

r 

r 

Work  with  measuring 
instrumems  (e.g., 
rulers) 

Work  with 

r 

r 

r 

r 

manipuiatives  (e.g., 
geometric  soAds) 

r 

r 

r 

r 

Use  a  calculator 

r 

r 

r 

r 

Use  a  computer 

r 

r 

r 

r 

Take  pracbce  tests 

r 

r 

r 

r 

Practice  by  teaching  or 
explairwng  to  each 
other 

r 

r 

r 

r 

Work  on  math  life 

skiHs"  such  as 
balarxsng  checkbooks, 
making  budgets, 
calculating  sale  prices, 
etc. 

r 

r 

r 

r 
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How  often  do  you  use  the  foUowing  in  your  class? 


Never  or 
hardly  ever 

A  few  times 

during  the 
course 

OrKe  or  twice  a 

month 

At  least  once  a 

Multiple  choice  tests 

r 

r 

r 

r 

Problem  sets 

r 

r 

r 

r 

Short  or  long  written 
responses 

r 

r 

r 

r 

Irxlividual  or  group 
projects  or  presentations 

- 

r 

At  the  beginning  of  the  program,  how  would  you  rate  your  students*  skills  in  each  of  the  following? 


No  students 

Few  students 

Sonne  students 

Most  students 

are  very 

are  very 

are  very 

are  very 

competere 

competent 

competent 

competent 

Addition  and  subtraction 

r 

r 

r 

r 

Multiplicabon 

r 

r 

r 

r 

Division 

r 

r 

r 

r 

Please  indicate  which  of  the  following  statements  is  nK>st  closely  aligned  with  your 
mathematics  teaching  philosophy  (neither  statement  may  represent  your  exact  philosophy,  but 
please  indicate  the  staten^ent  that  con'ies  closer  to  your  philosophy): 

Students  niust  be  taught  math  skWs  based  on  formulas  or  algorithms.  tnstructk>o  should  emphasize  procedural 
r  mathematics  and  provide  step-by-step  examples  with  skill  exercises.  Focus  should  be  placed  on  the  memorliation  of 
basic  math  facts  and  on  teaching  algorithms  that  efficiently  lead  to  the  correct  answer. 

Instruction  should  be  focused  on  the  process  of  leading  to  the  correct  answer,  rather  than  the  answer  itself.  Students 
are  encouraged  to  represent,  describe,  and  communicate  mathematical  ideas  by  a  variety  of  methods.  Conceptual 
understafMllng  is  the  primary  goal;  algoriihmic  fluency  will  follow  secondarily.  Students  should  learn  how  to  make 
connections  between  corrcepts. 
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English/Language  Arts  Specific  Questions 

Please  answer  the  following  questions  based  on  your  experiences  with  a  typical  English  dass.  such  as  your 
9am  dass.  For  example,  please  answer  time*based  questions  by  indicating  how  nr«ny  hours  per  week  you 
spend  on  a  specific  topic  or  subject  in  a  typical  dass.  If  class  variation  makes  it  difficult  to  choose  a  typical 
dass,  please  answer  the  questions  based  on  what  you  do  in  the  first  dass  you  teach  each  day. 


How  much  en^phasis  do  you  place  on  the  following? 


Not  covered 

Review  topic 

Minor  topic 

Major  topic 

Grammar 

r 

r' 

r 

r 

Uterature 

r 

r 

r 

r 

Composftion 

r 

r 

r 

r 

Reading 

comprenension 

r 

r 

r 

r 

Study  skJMs 

r 

r 

r 

r 

SpHlirvQ 

r 

r 

r 

r 
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How  often  do  yoti  use  the  following  types  of  literature? 


Never 

Rarely 

Some 

or  the  time 

Majority 

or  the  time 

FkUon 

r 

r 

r 

r 

Poetry 

r 

r 

r 

r 

Hythology/rolc 

tales 

r 

r 

r 

r 

Biography 

r 

r 

r 

r 

Drama 

r 

r 

r 

r 

Argument  and 
persuasion 

r 

r 

r 

r 

ProcedurU 

r 

r 

r 

r 

Other 

non-nction 

r 

In  addition  to  full-longth  selections  appeanng  in  any  textbook /workbook  you  n^ay  use,  how 
many  books  will  you  require  your  students  to  read  during  the  course  of  this  dass? 


How  often  do  you  do  the  following? 


Never  or 
hardly  ever 

Once  or  twice 

a  month 

OfKe  or  twice 

a  week 

Almost  every 
day 

Ask  a  student  to  read 

aloud 

r 

r 

r 

r 

Ask  students  to  write 

about  what  they  have 
read 

r 

r 

r 

r 

Give  students  time  to 

read  books  oT  their 

choice 

r 

r 

r 

r 

Ask  students  to  do  a 
group  adivty  about 
what  they  read 

r 

r 

r 

r 

Watch  movies  or  videos, 
listen  to  tapes  or  CDs 

r 

r 

r 

r 

Ask  students  to  make 
predictions  about  what 
they  are  reading  as  they 
are  reading 

r 

r 

r 

r 

Ask  students  to 

summarize  what  they 
have  read 

r 

r 

r 

r 
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To  what  extent  do  yoo  emphasize  the  following  processes  when  reading  informational  and 
literary  texts  in  dass? _ 


Lodte/recaM 

material 

Not  at  all 

r 

To  a  small 

extent 

r 

To  a  moOerate 

extent 

r 

To  a  large 
extent 

r 

Integrate/imtefpret 

material 

r 

r 

r 

r 

Crtique/evaluate 

material 

r 

r 

r 

r 

How  often  do  you  ask  your  students  to  write  for  each  of  the  following  purposes? 


Never  or 

A  few  Omes 
durirtg  the 

Once  or  twice  a 

At  least  or>ce  a 

hardly  ever 

course 

nrionth 

week 

Explain  (e.g..  provKie 

information  about  a  topic 

r 

r 

r 

r 

or  steps  in  a  process) 

Persuade  (e.g.,  convince 

someone  to  do 

r' 

r 

r 

r 

something) 

Convey  experience  (e.g.. 
wrte  about  personal  or 

r 

r 

imagined  experiences) 

Sum  marize/syntbesize 
material  they  have  read 

r 

r 

r 

r 

When  you  are  teaching  writing,  what  percentage  of  your  instructional  time  is  spent  on  each  of  the 
following? _ 


0-24% 

25-49% 

50-74% 

75-89% 

90^% 

Development  of  ideas 

r 

r 

r 

r 

r 

Organization  of  ideas 

EfTectiveriess  of 

r 

r 

r 

r 

r 

expression  (e.g., 
senterxe  variety,  word 
choice,  tone) 

r 

r 

c 

r 

r 

Nedianics  arxl  ^ 

conventions 

r 

r 

r 

r 

r 
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